Windows Server Essentials O365 Integration Errors


Here’s a quick post to describe an issue I didn’t see referenced anywhere else except for within forum replies.

Issue
A customer had Windows Server 2012 R2 Essentials configured with Office 365 Integration but noticed they were unable to make any changes to the integration (such as changing the Admin account or adding new users) and the Exchange Online-related status indicators in the Essentials Dashboard were not being displayed properly. The customer stated this issue happened once before but apparently resolved itself. However, in this case, functionality had been broken for several weeks before they decided to reach out to me.

Specifically, when running the O365 Integration wizard you would receive an error stating, “Cannot connect to Microsoft Online services…. Make sure that the computer is connected to the Internet and then try again.”

Resolution
I first looked under the C:\ProgramData\Microsoft\Windows Server\Logs folder within the SharedServiceHost-EmailProviderServiceConfig.log file for any Integration Tool errors.

The log revealed the following error messages:

BecWebServiceAdapter: Connect to BECWS failed due to known exception : System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at https://bws902-relay.microsoftonline.com/ProvisioningWebservice.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused

I was able to trace the error message to this Microsoft forum post where MVP Susan Bradley provided the resolution. In this case, the resolution was to navigate to the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Server\Productivity\O365Integration\Settings registry key and delete or rename the “BecEndPointAddress” registry entry. This entry had a value of the application server referenced in the error message above (bws902-relay.microsoftonline.com). After restarting all the Windows Server Essentials services, O365 Integration was fully restored.

My theory as to why this occurred is the application server the key referenced became inaccessible for some reason. I would think this value should be a load balanced name and not an individual server name.

Advertisements

Unable to logon to O365 via ADFS – ADFSAppPool stops (aka. I had a bad day)


Environment:
Customer using Exchange Online/Office 365 with no Exchange servers on-prem. Two ADFS 2.0 servers running on Server 2008 R2, enabling them to logon to Exchange Online via SSO (Single Sign On).

Issue:
After rebooting the two ADFS servers post Windows Updates the customer could no longer login to OWA & would receive a “503 Service Unavailable” error message via IIS on the two ADFS servers.

Background:
I have to hang my head in shame with this one as I really should have figured this out sooner. Initial troubleshooting showed that the ADFSAppPool was stopped in IIS. It would start but as soon as you tried accessing it, it would stop again. Nothing at all in the Application or ADFS logs in Event Viewer (more on this poor bit of troubleshooting on my part later).The ADFS service account it was running under looked ok; the App Pool would start & so would the ADFS Service (both running under this account) so it seemed to not be a credential issue (at least I got that part right). I even went as far as to reinstall ADFS & IIS on the non-primary ADFS server in the event it was something in IIS. I was clearly out-classed on this seemingly simple issue.

Resolution:
Because the customer was down & I was scratching my head, I decided to escalate the issue to Microsoft; at which point they resolved the issue in about 5min.

Now before I say the fix I’d just like to say I consider myself a good troubleshooter. I’ve been troubleshooting all manner of Microsoft, Cisco, etc technologies for more than a decade & made a pretty successful career out of it. I even managed to pass both the MCM 2010 & MCSM 2013 lab exams on the 1st attempt; but today was not my day. I spent over 2 hrs on this & I broke the cardinal rule of troubleshooting; I overlooked the simple things. Like many of us do I started digging a hole of deep troubleshooting, expecting this to be an incredibly complex issue; I was looking at SPN’s, SQL Permissions, checking settings in Azure, etc. I should have just looked back up in the sky instead of trying to dig a hole a mile deep but only 3 ft wide, because for some idiotic reason I chose to overlook the System Event logs….

I suppose once I saw nothing in the Application or ADFS logs I just moved on quickly to the next possibility but in a few short minutes the Microsoft Engineer checked the System Logs & saw Event 5021 from IIS stating that the service account did not have Batch Logon Rights (more on the event here). This lead him to look at Group Policy settings & sure enough, there was a GPO allowing only the Domain Admins group to log on as a batch job. (Reference 1 & 2). It seems this setting took effect after the ADFS servers were rebooted post Windows Updates. Not sure how the GPO got there as this solution was working for 2 years beforehand but it certainly was ruining our day today. After the GPO was modified to allow the ADFS service account to log on as a batch job, the issue was resolved after some service restarts.

Moral of the story:
Never overlook the obvious!
 It’s the best advice I can give to anyone, anywhere, & who has to troubleshoot anything. I’d like to say this is the 1st time this has happened to me but it’s not. Overlooking typos, not checking to see if a network cable is plugged in, not checking to see if a service is started… It happens to the best of us. I suppose overlooking the simple solution is just part of the human condition…..or at least whatever condition I have….. 🙂

Connecting to Lync Online


Issue when connecting to Lync Online you may receive one of the the following errors

“The ‘New-CsOnlineSession’ command was found in the module ‘LyncOnlineConnector’, but
the module could not be loaded. For more information, run ‘Import-Module LyncOnlineConnector’.”

and

“Unable to discover PowerShell endpoing URI
At C:\Program Files\Common Files\Microsoft Lync Server
2013\Modules\LyncOnlineConnector\LyncOnlineConnectorStartup.psm1”

 

Lets deal with the first one,:

First ensure you have downloaded and installed the Powershell Module for Lync Online

Next, and this is the weird part set the powershell execution policy to “unrestricted”, is seems there is an issue with the modules loading, we were able to discover this by comparing a powershell session for one system that worked and other that did not. (if you are security conscious you may want to set this back when you are done)
The specific command is set-ExecutionPolicy unrestricted

For the last error ensure you have configured your Lyncdiscover records in your External DNS, this is needed for powershell to detect and connect to your online Lync environment, if you are in a hybrid configuration you may be pointing this to on-prem Lync and not Lync online. you can find your specific configuration by clicking on your domain in the office 365 admin center.

Finally the syntax to connect to your Lync Online is:esco over anything in particular let

$LiveCred = Get-Credential
$LyncSession = New-CsOnlineSession -Credential $livecred
Import-PSSession $LyncSession

Creating Custom DLP Classification Rules and Policy


When at first I was looking into this the TechNet documentation was extensive and yet not as specific as I would prefer, so here is the quick and dirty DLP classification!

Creating and importing custom Classifications

  1. First you need to create your custom policy XML (Example Below)
  2. Save as XML Unicode file type (C:\MyNewPolicy.xml)
  3. Open the XML in internet explorer if its formatted correctly you will see the XML.
  4. Then import with Powershell
    New-ClassificationRuleCollection –FileData ([Byte[]]$(Get-Content -path C:\MyNewPolicy.xml -Encoding byte -ReadCount 0))
  5. Once its imported you should be able to create a new DLP policy using the EAC

Creating a custom DLP Rule

  1. Login to EAC (i.e https://mail.domain.com/ecp)
  2. Click Compliance Management, data loss prevention
  3. Click the Plusimage , then New custom policy
    image
  4. Name your policy and Choose your mode (I like to test with Policy tags), and click Save
    image
  5. Select the policy and click the image edit your new policy
  6. Select Rules from the left
  7. Click the imageto Create a new rule
  8. On the Apply this rule if field choose The message contains Sensitive information..
  9. Click *Select sensitive information types….. (if applicable)
  10. Click the imageto choose from the list,
  11. You should now see your new classification (from the example below it would be Secure Product Codes\ DLP by Exchangemasters.info)

image

Useful Tools

Example of a Rule Classification XML

 <?xml version=”1.0″ encoding=”utf-16″?>

 <RulePackage xmlns=”http://schemas.microsoft.com/office/2011/mce”&gt;

 <RulePack id=”b4b4c60e-2ff7-47b2-a672-86e36cf608be”>

  <Version major=”1″ minor=”0″ build=”0″ revision=”0″/>

  <Publisher id=”7ea13c35-0e58-472a-b864-5f2e717edec6″/>

  <Details defaultLangCode=”en-us”>

  <LocalizedDetails langcode=”en-us”>

  <PublisherName>DLP by Exchangemasters.info</PublisherName>

  <Name>Secure Product Codes</Name>

  <Description>Secure Products</Description>

  </LocalizedDetails>

  </Details>

  </RulePack>

  <Rules>

  <!– Product Code –>

  <Entity id=”acc59528-ff01-433e-aeee-13ca8aaee159″ patternsProximity=”300″ recommendedConfidence=”75″>

 <Pattern confidenceLevel=”75″>

  <IdMatch idRef=”Regex_Product_Code” />

  <Match idRef=”Code” />

  </Pattern>

  </Entity>

  <Regex id=”Regex_Product_Code”>[A-Z]{3}[0-9]{9}

  </Regex>

  <Keyword id=”Code”>

  <Group matchStyle=”word”>

  <Term>Code</Term>

  </Group>

  </Keyword>

  <LocalizedStrings>

  <Resource idRef=”acc59528-ff01-433e-aeee-13ca8aaee159″>

  <Name default=”true” langcode=”en-us”>

  Product Code

  </Name>

  <Description default=”true” langcode=”en-us”>

 A custom classification for detecting product codes that have 3 uppercase letters and 9 numbers

 </Description>

 </Resource>

</LocalizedStrings>

</Rules>

</RulePackage>

New behavior in Outlook 2013 causing certificate errors in some environments


Background:

I originally discovered this issue back in early Feb & let a couple people on the Exchange Product Team know about it via the TAP but it seems to be affecting more customers than initially thought so I thought I’d share.

In Outlook 2007 through Outlook 2010 all domain-joined Outlook clients would initially query Active Directory for AutoDiscover information & ultimately find a Service Connection Point (SCP) value that would point them to their nearest Client Access Server’s AutoDiscover virtual directory. If that failed then they would revert to using DNS like any non-domain-joined Outlook client. The order of this non-domain-joined lookup is as follows:

https://company.com/autodiscover/autodiscover.xml

https://autodiscover.company.com/autodiscover/autodiscover.xml

Local XML File

http://company.com/autodiscover/autodiscover.xml (looking for a redirect website)

SCP AutoDiscover Record

Why it ever looked to https://company.com/autodiscover/autodiscover.xml I’ll never really know because honestly I’ve never come across a customer who had it deployed that way; most have https://autodiscover.company.com/autodiscover/autodiscover.xml but I imagine when Exchange 2007 was first being developed they weren’t exactly sure how customers would be implementing AutoDiscover.

Issue:

The above methods have served us well since Exchange 2007 timeframe but for some reason the Outlook team decided to try & implement some giddyup into Outlook & try to speed up the process. They decided to have domain-joined Outlook 2013 clients query both the SCP values in AD as well as the DNS records at the same time. If an SCP record was found it would still be used but in the event it failed then it would already have the DNS response ready to go. Great idea, however there’s one problem in the implementation.

If Outlook 2013 encounters any kind of Certificate error while doing the simultaneous DNS query then you will receive a pop-up in Outlook about the cert.

I actually stumbled upon this while in the middle of the scenario below:

error

That’s right, I actually get a certificate pop-up for my lab’s domain name (ash15.com) & not autodiscover.ash15.com like one would expect if I were to have a certificate issue on Exchange.

When Outlook 2013 does it’s simultaneous DNS AutoDiscover query the first URL it tries is https://company.com/autodiscover/autodiscover.xml, which in my lab environment resolved to my Domain Controller, which was also serving DNS, as well as a Certificate Authority. Ash15.com resolved to this server because it’s my internal Active Directory domain name & the name server entry resolves to my DC (just ping internaldomainname.local in your AD lab environment & you’ll see the same thing).

Now because I have web enrollment enabled & am listening on 443 in IIS the server responded. Also, because I did not have a cert installed on the server with ash15.com in the Subject or Subject Alternative Name then it gave the certificate error we see above.

Resolution:

The error is easy enough to get through & it only occurred on initial profile creation but this can definitely prove painful for some customers. Obviously my lab environment is a corner case but there have been several other customers report this issue with Outlook 2013 as well.

Here’s an example scenario.

Imagine you have a public website for andrewswidgets.com hosted by a third-party hosting site & you did not pay for HTTPS/443 services. However if you were to query the website using https then it could respond & obviously not return a certificate with andrewswidgets.com on it (because you haven’t paid for it you cheapskate…). Now imagine you begin deploying users using Outlook 2013 in your internal environment. In the past, they would have found the SCP record that would have pointed them to your internal Exchange 07/10/13 server for AutoDiscover & would have been happy as a clam (one Exchange Product Manager’s favorite way to describe Exchange bliss). However, now they may get a certificate pop-up for andrewswidgets.com when creating a new profile.

There are a couple ways around this. Make sure andrewswidgets.com doesn’t listen on 443, or possibly get a proper cert on your website that is listening on 443. Simply put, just make sure whatever andrewswidgets.com resolves to is something that’s not going to throw a certificate error.

I’ve heard nothing concrete or public but the Outlook team is aware of the issue & listening to customer feedback. I suggest contacting Microsoft Support if your organization is running into this issue.

 

Also, this KB offers methods to control which AutoDiscover methods are used by your Outlook clients

 

Configure your target URL for OWA redirect when migrating users to the cloud


 

When you migrate  a user to Office 365  you want OWA users to have a simple redirect to office 365 and not get this error:

image

Also you want to give your users an easy OWA url not http://outlook.com/owa/mysupercooldomain.com

The solution is 2 steps

  1. create a cname record that points to outlook.com ( i.e. OWA.mysupercooldomain.com = outlook.com)
  2. add that record to your organization relationship
    1. set-orginaizationrelationship –targetOwaUrl http://owa.mysupercooldomain.com/owa
  3. Give OWA.mysupercooldomain.com to your users as there new owa page

Note: the domain you create the CNAME in must be one of your federated or accepted domains in office 365 for realm discovery to work.

Problems with Federation Trust After changes to your certificate


Here is the situation and the solution

Situation

  • I Had a federated trust setup in exchange 2010 SP1 (same issue can happen in RTM)
  • I created it using the “UseLegacyProvisioningService” switch and so was using a 3rd party certificate
  • After the trust was established I had some issues with the cert… and while it’s a long story the gist is that the cert was revoked and I received a new one.
  • Well this caused an issue with my federation trust because I didn’t get the cert switched before the revocation (this can also happen if you delete the cert from the cert store or if it expires before you roll to a new one)

Symptoms: I received the following errors when I try to make any changes to the Federation trust or even try to delete it.

    An error occurred accessing Windows Live. Detailed information: "The request failed with HTTP status 403: Forbidden.".
        + CategoryInfo          : InvalidResult: (:) [Set-FederationTrust], LiveDomainServicesException
        + FullyQualifiedErrorId : 84DE3E74,Microsoft.Exchange.Management.SystemConfigurationTasks.SetFederationTrust

    Error:
    Exception has been thrown by the target of an invocation.

     

    An error occurred accessing Windows Live. Detailed information: "The request failed with HTTP status 403: Forbidden.".

    The request failed with HTTP status 403: Forbidden.
    Click here for help… http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.1.218.11&t=exchgf1&e=ms.exch.err.Ex1FCF67

Reason: the certificate that was used and is expected is no longer valid and so cannot be trusted on the live servers at Microsoft

Solution: Use ADSIEdit to change the cert to the new thumbprint

  1. Add the new cert as the next cert in EMC under Federation Trusts
  2. Open ADSIEDit with Domain admin Credentials
  3. Connect to Configuration naming context
  4. Browse to Domain –> Configuration –> Services –> Microsoft Exchange –> OrgName –> Federation Trusts
  5. image
  6. Rich Click on your Federation Trust in the right hand window and go to properties
  7. Scroll down until you find the key “msExchFedOrgNextPrivCertificate” (this was where my solution varied from EXPTA’s)
  8. Edit the key and select all the contents and copy, then close the key
  9. Edit the Key “msExchFedOrgPrivCertificate” and paste in your copied contents (It may be a good idea to have a copy of this keys contents before overwriting it)
  10. Close all windows
  11. re-open the EMC or EMS run your failed commands again and life is grand!

Thanks to EXPTA and Gene at Microsoft for the assist in figuring this out.