Web Management Service will not start and causes Exchange update to fail


Today I had an Exchange update issue that I’d previously never encountered before. Exchange 2013 CU10 update failed saying the Web Management Service could not be started. Attempts to manually start the service failed. Application logs pointed to IIS-IISManager 1007 event saying the following:

“Unable to read the certificate with thumbprint ‘{thumbprint}’. Please make sure the SSL certificate exists and that is correctly configured in the Management Service page.”

The thumbprint it was listing was not found on the server, either using Get-ExchangeCertificate or the MMC certificate snap-in. A web search led me to the below article which resolved the issue. Normally, an Exchange server will have a certificate called “WMSvc-servername” (Friendly Name of WMSvc) and it will be bound in IIS to the Web Management Service, but in this case the certificate was missing. By binding another certificate to the service we were able to get the service to start and continue the Exchange Update. An alternative would be to request a new certificate for the purposes of this service.

https://technet.microsoft.com/en-us/library/cc735088(v=ws.10).aspx

Find the SSL certificate that the Web Management Service is using

To find the SSL certificate that the Web Management Service is using:

  1. Click Start, click Control Panel, and then click Administrative Tools.
  2. Right-click Internet Information Services (IIS) Manager and select Run as administrator.
  3. In the Connections pane, select the server that you want to manage.
  4. In Features View, double-click Management Service.
  5. Under SSL certificate ensure that a certificate is selected.
  6. Note the name of the certificate. By default, the name starts with “WMSvc”.

Additional Reference:
http://exctech2013.blogspot.com/2013/10/the-web-management-service-could-not-be.html

Advertisements

Emails from scanner to Exchange 2013 being sent as separate attachment


Scenario

After switching from hosted email to Exchange 2013 on-premises, a customer noticed that when using scan-to-email functionality the .PDF files it created were not showing up as expected. Specifically, instead of an email being received with the .PDF attachment of the scanned document, they were receiving the entire original message as an attachment (which then contained the .PDF).

When the scanner was configured to send to an external recipient (Gmail in this case), the issue did not occur & the message was formatted as expected. The message was still being relayed through Exchnage, it was just the recipient that made the difference. See the below screenshots for examples of each:

What the customer was seeing (incorrect format)

A

What the customer expected to see (correct format)

B

This may not seem like a big issue but it resulted in users on certain mobile devices not being able to view the attachments properly.

Troubleshooting Steps

There were a couple references on the MS forums to similar issues with older versions of 2013, but this server was updated. My next path was to see if there were any Transport Agents installed that could’ve been causing these messages to be modified. I used many of the steps in my previous blog post “Common Support Issues with Transport Agents” including disabling two 3rd party agents & restarting the Transport Service; the issue remained.

My next step was to disable both of the customer’s two Transport Rules (Get-TransportRule | Disable-TransportRule); one was related to managing attachment size while the other appended a disclaimer to all emails. This worked! By process of elimination I was able to determine it was the disclaimer rule causing the messages to be modified.

Resolution

Looking through the settings of the rule the first thing that caught my eye was the Fallback Option of “Wrap”. Per this article from fellow MVP Pat Richard, Wrap will cause Exchange to attach the original message & then generate a new message with our disclaimer in it (sounds like our issue).

C

However, making this change did not fix the issue, much to my bewilderment. There seemed to be something about the format of the email that Exchange did not like; probably caused by the formatting/encoding the scanner was using.

Ultimately, the customer was fine with simply adding an exception to the Transport Rule stating to not apply the rule to messages coming from the scanner sender email address.

D

 

Common Support Issues with Transport Agents


This is a fairly basic post but it happens enough that I’d like to call out the basics of troubleshooting it. I’ve seen many cases over time where mail flow is either being halted or become sluggish due to a third-party transport agent (I actually saw 3 instances of this happening this past month which prompted this post).

Examples of Transport Agents could be Anti-Virus software, Anti-Spam software, DLP software, agents which add disclaimers to email messages, or email archiving solutions. I won’t call out specific vendors as I don’t think there’s necessarily anything wrong with any particular one. Sometimes an install of a piece of software just becomes corrupted or there’s some unforeseen incompatibility between the third-party software & Exchange; or some other software in the environment. However, sometimes the Agent can indeed have a bug which needs to be addressed with the vendor.

Anyways, here’s the ways in which I’ve seen these issues manifest themselves:

  • Messages Stuck in the Submission queue
  • A delay in SMTP response (when you telnet to the Exchange Server over 25, it takes longer than expected for the server’s SMTP banner to be displayed)
  • Messages are slow to flow through the transport pipeline (general slow delivery)
  • Microsoft Exchange Transport Service will not start or repeatedly crashes

To highlight more recent examples, last week I had a colleague come to me saying he had two Exchange 2010 Hub/CAS boxes, with the same config, yet one of them would have a slower connection when he would telnet to it; the banner would take at least 20 seconds to be displayed. This also resulted in the health checks for the hardware load balancer in place to mark the server as down. Each server had the same Anti-V/Anti-SPAM software installed, yet only one was showing the symptoms. For testing purposes he “disabled” the third-party software using its management interface but the issue persisted.

However, after running a “Get-TransportAgent” on the server, the Transport Agent still showed as being “Enabled”. This demonstrates a point I frequently make with customers, that disabling Anti-Virus software rarely serves as a useful troubleshooting step (even file-based Anti-V). This is because the TransportAgent is typically still enabled. For file-based Anti-Virus, even with the Services disabled there is usually still a network filter driver that is sitting on the TCP/IP stack which could be causing issues (only an uninstall of the 3rd-party product removes it).

Bottom-line, an uninstall is still the best method to remove potentially problematic Anti-V/Anti-SPAM/Anti-Malware software. So in this case the issue was a bad/corrupted install of the product on that server.

Another scenario (also Exchange 2010) was where messages were stuck in the Submission Queue for extended periods of time. The Application Logs were filled with Event 1050 MSExchange Extensibility events which were stating the installed agent was taking an unusual amount of time to process an event; thus causing the delay in transport (Reference 1 2 3).

After running Get-TransportAgent I was actually greeted by an error message saying it was unable to access a file located in the “C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\agents” directory. This is where the files associated with your Transport Agents are stored. So again, the issue was a corrupted install of the product. Reinstalling the software resolved the issue.

So nothing fancy about this one. Just check Event Viewer for Transport events or use process of elimination if you’re experiencing any of the symptoms above. Having worked with Microsoft Support many times in the past, they will almost always ask you to remove third-party components such as Anti-V if they are unable to pinpoint the issue to its source; so save yourself some time & rule it out first.

I know some people work for companies where this is like pulling teeth but it’s always going to be a battle between usability & security. If your management requires you spend 40 hours on the phone working with a vendor or Microsoft before finally being told you’re going no further until removing the third-party component then I give you my best & suggest you get the coffee started. We all know the most important acronym in IT is CYA after all 😉

For great reading on Exchange Transport Agents see MCM/MCSM/MVP Brian Reid’s two posts on the topic

Creating a Simple Exchange Server Transport Agent

Exchange 2013 Transport Agents

New behavior in Outlook 2013 causing certificate errors in some environments


Background:

I originally discovered this issue back in early Feb & let a couple people on the Exchange Product Team know about it via the TAP but it seems to be affecting more customers than initially thought so I thought I’d share.

In Outlook 2007 through Outlook 2010 all domain-joined Outlook clients would initially query Active Directory for AutoDiscover information & ultimately find a Service Connection Point (SCP) value that would point them to their nearest Client Access Server’s AutoDiscover virtual directory. If that failed then they would revert to using DNS like any non-domain-joined Outlook client. The order of this non-domain-joined lookup is as follows:

https://company.com/autodiscover/autodiscover.xml

https://autodiscover.company.com/autodiscover/autodiscover.xml

Local XML File

http://company.com/autodiscover/autodiscover.xml (looking for a redirect website)

SCP AutoDiscover Record

Why it ever looked to https://company.com/autodiscover/autodiscover.xml I’ll never really know because honestly I’ve never come across a customer who had it deployed that way; most have https://autodiscover.company.com/autodiscover/autodiscover.xml but I imagine when Exchange 2007 was first being developed they weren’t exactly sure how customers would be implementing AutoDiscover.

Issue:

The above methods have served us well since Exchange 2007 timeframe but for some reason the Outlook team decided to try & implement some giddyup into Outlook & try to speed up the process. They decided to have domain-joined Outlook 2013 clients query both the SCP values in AD as well as the DNS records at the same time. If an SCP record was found it would still be used but in the event it failed then it would already have the DNS response ready to go. Great idea, however there’s one problem in the implementation.

If Outlook 2013 encounters any kind of Certificate error while doing the simultaneous DNS query then you will receive a pop-up in Outlook about the cert.

I actually stumbled upon this while in the middle of the scenario below:

error

That’s right, I actually get a certificate pop-up for my lab’s domain name (ash15.com) & not autodiscover.ash15.com like one would expect if I were to have a certificate issue on Exchange.

When Outlook 2013 does it’s simultaneous DNS AutoDiscover query the first URL it tries is https://company.com/autodiscover/autodiscover.xml, which in my lab environment resolved to my Domain Controller, which was also serving DNS, as well as a Certificate Authority. Ash15.com resolved to this server because it’s my internal Active Directory domain name & the name server entry resolves to my DC (just ping internaldomainname.local in your AD lab environment & you’ll see the same thing).

Now because I have web enrollment enabled & am listening on 443 in IIS the server responded. Also, because I did not have a cert installed on the server with ash15.com in the Subject or Subject Alternative Name then it gave the certificate error we see above.

Resolution:

The error is easy enough to get through & it only occurred on initial profile creation but this can definitely prove painful for some customers. Obviously my lab environment is a corner case but there have been several other customers report this issue with Outlook 2013 as well.

Here’s an example scenario.

Imagine you have a public website for andrewswidgets.com hosted by a third-party hosting site & you did not pay for HTTPS/443 services. However if you were to query the website using https then it could respond & obviously not return a certificate with andrewswidgets.com on it (because you haven’t paid for it you cheapskate…). Now imagine you begin deploying users using Outlook 2013 in your internal environment. In the past, they would have found the SCP record that would have pointed them to your internal Exchange 07/10/13 server for AutoDiscover & would have been happy as a clam (one Exchange Product Manager’s favorite way to describe Exchange bliss). However, now they may get a certificate pop-up for andrewswidgets.com when creating a new profile.

There are a couple ways around this. Make sure andrewswidgets.com doesn’t listen on 443, or possibly get a proper cert on your website that is listening on 443. Simply put, just make sure whatever andrewswidgets.com resolves to is something that’s not going to throw a certificate error.

I’ve heard nothing concrete or public but the Outlook team is aware of the issue & listening to customer feedback. I suggest contacting Microsoft Support if your organization is running into this issue.

 

Also, this KB offers methods to control which AutoDiscover methods are used by your Outlook clients

 

What version is my Exchange Server?


Have you run into an issue where you were asked what build of exchange you have and weren’t able to provide a quick answer? well now you can.

 

Here is the Microsoft Wiki that contains all the 2007-2010 version numbers.

http://social.technet.microsoft.com/wiki/contents/articles/exchange-server-and-update-rollups-builds-numbers.aspx

For older versions http://support.microsoft.com/kb/158530

 

To find your build number just run this from powershell

Get-ExchangeServer |ft identity,*DisplayVersion

Exchange 2007/2010 certificates and new Go Daddy


I found recently a situation where the Godaddy cert chain wasn’t installed and some phone clients had issues, the following is a proven request\import process to install the cert and chain.

 So I gave this process to a friend and found out that I should either rename or subtitle this post, “Getting SBS 2008 to recognige your 2048 bit certificate” – great side effect!

  1. First make your cert request in powershell, it should look something like this:
    • New-ExchangeCertificate -GenerateRequest -SubjectName “C=US, O=Company, CN=mail.domain.com” -domainname mail.domain.com,autodiscover.domain.com,hostname,hostname.domain.local -FriendlyName mail.domain.com -privatekeyexportable:$true -path c:\cert_myserver.txt
    • IMPORTANT: don’t do any new cert requests or run any wizards until the cert is imported
  2. Send the cert request to Godaddy as a UCC certificate
  3. Import the Certificate to complete the request
    • Import-ExchangeCertificate –Path “C:\CertificateFile.cer” | Enable-ExchangeCertificate -Services pop, smtp, iis, imap
  4. Export the certificate
    1. Start –> Run –> MMC –> Add Snap-in –> certificates –> Local computer
    2. Right click certificate –> all tasks –> export
      1. Include the certificate chain and private key
      2. Enter a password
  5. Re-Import certificate including chain (this imports the chain certs also)
    1. Right click in a blank area of the certificate MMC –> all tasks –> import
    2. Select the certificate you exported –> import –> include the certificate chain.

 

And yes there are other was to get to the same result, this is just a simple easy to explain way

Re-create Exchange 2007 OWA Virtual Directories..


Have you had issues with OWA where you needed to re-create the directories? or maybe you made customizations and now it doesn’t work and you want to just get back to the default config?

Here are some scripts to get re-install the OWA virtual directories, they have been tested in lab environments

As with everything use at your own risk and always make a backup first!
YOU are responsible for your server\data.

ONLY do this if you have 1 CAS server and have not customized the OWA directories.

Save this as rebuildOWA.ps1 and run from powershell

************************Start of script**********************************************

$server = hostname

Get-OwaVirtualDirectory -server $server | Remove-OwaVirtualDirectory

New-OwaVirtualDirectory -name “owa” -OwaVersion Exchange2007 -WebSiteName “Default Web Site”

New-OwaVirtualDirectory “exchange” -OwaVersion Exchange2003or2000 -VirtualDirectoryType Mailboxes -WebSiteName “Default Web Site”

New-OwaVirtualDirectory “public” -OwaVersion Exchange2003or2000 -VirtualDirectoryType PublicFolders -WebSiteName “Default Web Site”

New-OwaVirtualDirectory “exchweb” -OwaVersion Exchange2003or2000 -VirtualDirectoryType Exchweb -WebSiteName “Default Web Site”

new-owavirtualdirectory “Exadmin” -owaversion:Exchange2003or2000 -virtualDirectoryType Exadmin -WebSiteName “Default Web Site”

Get-AutodiscoverVirtualDirectory | Remove-AutodiscoverVirtualDirectory

 New-AutodiscoverVirtualDirectory -WebsiteName “Default Web Site” -BasicAuthentication $true -WindowsAuthentication $true

IISRESET

************************End of script********************************************

Here is one for SBS 2008

Save this as rebuildSBSOWA.ps1 and run from powershell

************************Start of script**********************************************

$LocalServerName = hostname

$ActiveSyncMailboxName = “Windows SBS Mobile Mailbox Policy” + ” ” + $LocalServerName

$OABVDir = $LocalServerName + “\OAB (SBS Web Applications)”

$OAB = Get-OfflineAddressBook | Select-Object -Property Name

$strDomainDNS = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain().Name

$DomainAdmins = $strDomainDNS + “\Domain Admins”

$OrgName = Get-OrganizationConfig | Select-Object -Property DistinguishedName

$DefaultExchangeCertificate = “CN=” + $LocalServerName

Get-ExchangeCertificate | Where { $_.Subject -eq “$DefaultExchangeCertificate” } | ForEach { Remove-ExchangeCertificate -Thumbprint $_.Thumbprint }

Get-OwaVirtualDirectory -server $LocalServerName | Remove-OwaVirtualDirectory

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2007” -ExternalAuthenticationMethods Fba

Set-OWAVirtualDirectory -InternalUrl “https://sites/owa/” -ClientAuthCleanupLevel “Low” -LogonFormat “UserName” -DefaultDomain $strDomainDNS -Identity “Owa (SBS Web Applications)”

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “Exadmin” -ExternalAuthenticationMethods Fba

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “Mailboxes” -ExternalAuthenticationMethods Fba

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “Exchweb” -ExternalAuthenticationMethods Fba

New-OWAVirtualDirectory -WebsiteName “SBS Web Applications” -OwaVersion “Exchange2003or2000” -VirtualDirectoryType “PublicFolders” -ExternalAuthenticationMethods Fba

iisreset /noforce

cd $env:windir\system32\inetsrv

.\appcmd.exe unlock config “-section:system.webserver/security/authentication/windowsauthentication”

.\appcmd.exe set site “Default Web Site” /Bindings:http/*:80:

.\appcmd.exe start site “Default Web Site”

.\appcmd.exe start site “SBS Web Applications”

************************End of script********************************************