Missing Recipient Creation Permissions in ECP and Shell

Issue
Customer running Exchange 2016 RTM was unable to create Mail Contacts in the Exchange Control Panel (ECP) or Exchange Management Shell (EMS). When opening EMS, the New-MailContact cmdlet was not visible, which was an indicator that there was a Role-Based Access Control (RBAC) permissions issue. Within ECP, the Plus symbol was not visible.

 

 

Troubleshooting steps taken
Customer was using the Administrator account that was a member of the “Organization Management” RBAC default role group. We also tested with a new account that was also added to the “Organization Management” group and the issue persisted. We also experimented with adding the test account to all default RBAC role groups and the issue still persisted. We also had the customer update to Cumulative Update 6 and the issue still persisted.

At this point, I wanted to verify the cmdlets were still present and available on the system (trying to rule out an odd corruption issue) so I opened PowerShell and manually loaded the Exchange module (bypassing RBAC) by running the following command:

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn

The New-MailContact cmdlet was available and working. Therefore, I knew it was an issue with RBAC via Remote PowerShell (as locally loading the Exchange modules bypasses RBAC).

I ran the below command (Get-ManagementRoleEntry) to view which management roles were associated with the New-MailContact cmdlet:

Get-ManagementRoleEntry *\New-MailContact | Format-List

From the output I could see that the “Mail Recipient Creation” role contains this cmdlet (Management Role Entry is synonymous with cmdlet within the RBAC framework). I then ran Get-ManagementRoleAssignment cmdlet to  view all of the default assignments. I compared these to my lab environment and they all appeared correct. Therefore, I decided to test a manual assignment of the “Mail Recipient Creation” role to my test account by running the below command (New-ManagementRoleAssignment):

New-ManagementRoleAssignment –Name TestFix –Role “Mail Recipient Creation” –User TestUser

After logging into ECP with the TestUser account, I could now create mail contacts. After opening EMS with TestUser I could now view New-MailContact as an available cmdlet:

 

 

This told me that somewhere in the RBAC framework, the Mail Recipient Creation role was missing as an assigned role for the default role groups.

Resolution
I was unfortunately unable to find the root cause and due to time constraints we chose to implement a workaround by manually adding the Mail Recipient Creation role to the Organization Management and Recipient Management role groups by running the below commands:

New-ManagementRoleAssignment –Name OrgMgmtFix –Role “Mail Recipient Creation” –SecurityGroup “Organization Management”

New-ManagementRoleAssignment –Name RcptMgmtFix–Role “Mail Recipient Creation” –SecurityGroup “Recipient Management”

After running these commands and logging into ECP again with the Administrator account, we were then able to create Mail Contacts.

Mailbox Anchoring affecting new deployments & upgrades

Update2 (March 1st 2016): Microsoft has released the following blog post which states this behavior will be reverted/absent in 2013 CU12 and RTM/CU1 versionf of Exchange 2016 Remote PowerShell Proxying Behavior in Exchange 2013 CU12 and Exchange 2016

Update: Microsoft has released the following KB article to address this issue: “Cannot process argument transformation” error for cmdlets in Exchange Server 2013 with CU11

Note: This article should also apply when Exchange 2016 CU1 releases and includes Mailbox Anchoring (unless Microsoft makes a change to behavior before it’s release). So the scenario of installing the first Exchange 2016 server using CU1 bits into an existing environment would also apply.

Summary

It was announced in Microsoft’s recent blog post about Exchange Management Shell and Mailbox Anchoring that the way Exchange is managed will change going forward. Starting with Exchange 2013 CU11 (released 12/10/2015) and Exchange 2016 CU1 (soon to be released), an Exchange Management Shell session will be directed to the Exchange Server where the user who is attempting the connection’s mailbox is located. If the connecting user does not have a mailbox, an arbitration mailbox (specifically SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c) will be used instead. In either case, if the mailbox is unavailable (because it’s on a database that’s dismounted or is on a legacy version of Exchange) then Exchange Management Shell will be inoperable.

Issue

While it has always been recommended to move system and Arbitration mailboxes to the newest version of Exchange as soon as possible, there is a scenario involving Exchange 2013 CU11 which have led to customer issues:

• Existing Exchange 2010 Environment
• The first version of Exchange 2013 installed into the environment is CU11
• Upon installation, the Exchange Admin is unable to use Exchange Management Shell on Exchange 2013. Thus preventing the management of Exchange 2013 objects
• The Exchange Admin may also be unable to access the Exchange Admin Center using traditional means

This is due to the new Mailbox Anchoring changes. If the Exchange Admin’s mailbox (or the Arbitration mailbox, if the Exchange Admin did not have a mailbox) was on Exchange 2013 then this issue would not exist. However, because this was the first Exchange 2013 server installed into the environment, and it was CU11, there was no way to prevent this behavior.

This issue was first reported by Exchange MVP Ed Crowley, and yesterday a customer of mine also encountered the issue. The symptoms were mostly the same but the ultimate resolution was fairly straightforward.

Possible Resolutions

Resolution#1:

Attempt to connect to Exchange Admin Center on 2013 using the “Ecp/?ExchClientVer=15” string at the end of the URL (Reference). For Example:

• HTTPS://2013ServerName/ecp/?ExchClientVer=15

I’ve heard mixed results using this method. When Ed Crowley encountered this issue, this URL worked, yet when I worked with my customer I was still unable to access EAC by using this method. However, it is worth an attempt. Once you’re connected to EAC, you can use it to move your Exchange Admin mailbox to 2013. However, should you not have a mailbox for your Exchange Admin account, this method may fail because there’s currently no way to move Arbitration Mailboxes via the EAC. So it’s recommended to create a mailbox for your Exchange Admin account using the EAC and then you’ll be able to connect via EMS.

Resolution#2:

Note: Using this method has a low probability of success as Microsoft recommends using the newer version of Exchange to “pull” a mailbox from the older version. Based on feedback I’ve received from Microsoft Support, you may consider just skipping this step and going to Step 3.

Use Exchange 2010 to attempt to move the Exchange Admin mailbox to a database on Exchange 2013. Historically, it’s been recommended to always use the newest version of Exchange to perform a mailbox move. In my experience this is hit or miss depending on the version you’re moving from and the version you’re moving to. However, it’s worth attempting:

Issue the below command using Exchange 2010 Management Shell to move the Exchange Admin’s mailbox to the Exchange 2013 server:

New-MoveRequest <AdminMailbox> -TargetDatabase <2013Database>

If the Exchange Administrator does not have a mailbox, then move the Arbitration mailboxes to Exchange 2013:

Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase <2013Database>

Resolution#3:

Connect to Exchange 2013 CU11 using Local PowerShell and manually load the Exchange modules:

• On the Exchange 2013 CU11 Server, open a Windows PowerShell window as Administrator
• Run the following command:
  • Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn

At this point the local PowerShell module can be used to move the Exchange Admin’s mailbox to the Exchange 2013 server:

New-MoveRequest <AdminMailbox> -TargetDatabase <2013Database>

If the Exchange Administrator does not have a mailbox, then move the Arbitration mailboxes to Exchange 2013:

Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase <2013Database>

In addition, there have been reported issues with 2013 EMS still having connectivity issues even after the relevant mailboxes have been moved. A different Windows user with appropriate Exchange permissions (using a different Windows profile) will work fine however. It seems there are PowerShell cookies for the initial profile used which could still be causing problems. In this scenario, you may have to remove all listed cookies in the following registry key (Warning, edit the registry at your own risk. A backup of the registry is recommended before making modifications):

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WSMAN\Client\ConnectionCookies

Summary

It should be noted that while this scenario involved Exchange 2013 CU11 being installed into an existing Exchange 2010 environment, it can affect other scenarios as well:

• Exchange 2013 CU11 or newer being installed into an existing Exchange 2010 environment
• Exchange 2013 CU11 or newer being installed into an existing Exchange 2007 environment
• Exchange 2016 CU1 (when released) or newer being installed into an existing Exchange 2010 environment

So unless Microsoft changes the behavior of Mailbox Anchoring, this is a precaution that should be taken when installing the first Exchange 2013 CU11/2016 CU1 (when released) server into an existing environment.

 

Edit: This forum post also describes the issue. In it, the user experiences odd behavior with the 2013 servers not being displayed if you run a Get-ExchangeServer & other odd behavior. This is similar to what I experienced in some lab testing. Ultimately, the same resolution applies.

https://social.technet.microsoft.com/Forums/en-US/05897b40-0717-437d-90ca-d550e3226c2a/exchange-2013-cu-11-breaks-some-admin-accounts-?forum=exchangesvrdeploy

 

Exchange Shell errors after incorrectly modifying IIS

Scenario
Customer stated that after replacing a certificate for their Exchange 2013 server they were unable to access Exchange Management Shell. The following error was displayed in Exchange Management Shell:

VERBOSE: Connecting to server-a.domain.com.
New-PSSession : [server-a.domain.com] Connecting to remote server server-a.domain.com failed with the following error message
: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not
available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more
information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI “$connectionUri” -ConfigurationName Microsoft.Excha …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotingTransportException
+ FullyQualifiedErrorId : URLNotAvailable,PSSessionOpenFailed

 

Resolution
In this case I decided to just refer to my own notes from a previous blog post. Because this error is typically associated with IIS related issues such as improper bindings, stopped web sites, or firewalls I made my way through each of the settings.

After right-clicking each of the web sites & selecting “Edit Bindings” I was greeted by the below image which immediately told me what was wrong.

(These images are actually from my lab where I recreated the issue)

It seems that in their confusion, instead of just using EAC or Exchange Management Shell to replace their certificate they decided to go into the default bindings (which rarely ever need to be modified using the IIS management tools) & add the subject name of their new certificate to the “Host Name” field of each binding. This was done on both the “Default Web Site” as well as the “Exchange Back End” website.

It’s certainly unnecessary but while it may seem harmless, it actually negatively affected the way in which IIS handles the incoming client connections. Since the Exchange Management Shell module sends the request using the Exchange Server’s internal FQDN, IIS would not answer the request because to it, it was not hosting that service. It was only answering requests for mail.ash.com (my lab’s name for the purpose of issue reproduction in this article). Interestingly enough, we could access OWA/ECP etc. using mail.ash.com but we were unable to access those services using the server’s hostname/FQDN. This makes perfect sense if you consider how IIS treats inbound connections when you use Host Names to define binding. Simply put, if you don’t leave blank Host Name fields, IIS will only answer requests for the Host Names you specifically defined.

So the solution was to blank out the Host Names & restart IIS. After doing so EMS connected without issue.