Mailbox Anchoring affecting new deployments & upgrades


Update2 (March 1st 2016): Microsoft has released the following blog post which states this behavior will be reverted/absent in 2013 CU12 and RTM/CU1 versionf of Exchange 2016 Remote PowerShell Proxying Behavior in Exchange 2013 CU12 and Exchange 2016

Update: Microsoft has released the following KB article to address this issue: “Cannot process argument transformation” error for cmdlets in Exchange Server 2013 with CU11

Note: This article should also apply when Exchange 2016 CU1 releases and includes Mailbox Anchoring (unless Microsoft makes a change to behavior before it’s release). So the scenario of installing the first Exchange 2016 server using CU1 bits into an existing environment would also apply.

Summary

It was announced in Microsoft’s recent blog post about Exchange Management Shell and Mailbox Anchoring that the way Exchange is managed will change going forward. Starting with Exchange 2013 CU11 (released 12/10/2015) and Exchange 2016 CU1 (soon to be released), an Exchange Management Shell session will be directed to the Exchange Server where the user who is attempting the connection’s mailbox is located. If the connecting user does not have a mailbox, an arbitration mailbox (specifically SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c) will be used instead. In either case, if the mailbox is unavailable (because it’s on a database that’s dismounted or is on a legacy version of Exchange) then Exchange Management Shell will be inoperable.

Issue

While it has always been recommended to move system and Arbitration mailboxes to the newest version of Exchange as soon as possible, there is a scenario involving Exchange 2013 CU11 which have led to customer issues:

  • Existing Exchange 2010 Environment
  • The first version of Exchange 2013 installed into the environment is CU11
  • Upon installation, the Exchange Admin is unable to use Exchange Management Shell on Exchange 2013. Thus preventing the management of Exchange 2013 objects
  • The Exchange Admin may also be unable to access the Exchange Admin Center using traditional means

This is due to the new Mailbox Anchoring changes. If the Exchange Admin’s mailbox (or the Arbitration mailbox, if the Exchange Admin did not have a mailbox) was on Exchange 2013 then this issue would not exist. However, because this was the first Exchange 2013 server installed into the environment, and it was CU11, there was no way to prevent this behavior.

This issue was first reported by Exchange MVP Ed Crowley, and yesterday a customer of mine also encountered the issue. The symptoms were mostly the same but the ultimate resolution was fairly straightforward.

Possible Resolutions

Resolution#1:

Attempt to connect to Exchange Admin Center on 2013 using the “Ecp/?ExchClientVer=15” string at the end of the URL (Reference). For Example:

I’ve heard mixed results using this method. When Ed Crowley encountered this issue, this URL worked, yet when I worked with my customer I was still unable to access EAC by using this method. However, it is worth an attempt. Once you’re connected to EAC, you can use it to move your Exchange Admin mailbox to 2013. However, should you not have a mailbox for your Exchange Admin account, this method may fail because there’s currently no way to move Arbitration Mailboxes via the EAC. So it’s recommended to create a mailbox for your Exchange Admin account using the EAC and then you’ll be able to connect via EMS.

Resolution#2:

Note: Using this method has a low probability of success as Microsoft recommends using the newer version of Exchange to “pull” a mailbox from the older version. Based on feedback I’ve received from Microsoft Support, you may consider just skipping this step and going to Step 3.

Use Exchange 2010 to attempt to move the Exchange Admin mailbox to a database on Exchange 2013. Historically, it’s been recommended to always use the newest version of Exchange to perform a mailbox move. In my experience this is hit or miss depending on the version you’re moving from and the version you’re moving to. However, it’s worth attempting:

Issue the below command using Exchange 2010 Management Shell to move the Exchange Admin’s mailbox to the Exchange 2013 server:

New-MoveRequest <AdminMailbox> -TargetDatabase <2013Database>

If the Exchange Administrator does not have a mailbox, then move the Arbitration mailboxes to Exchange 2013:

Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase <2013Database>

Resolution#3:

Connect to Exchange 2013 CU11 using Local PowerShell and manually load the Exchange modules:

  • On the Exchange 2013 CU11 Server, open a Windows PowerShell window as Administrator
  • Run the following command:
    • Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn

At this point the local PowerShell module can be used to move the Exchange Admin’s mailbox to the Exchange 2013 server:

New-MoveRequest <AdminMailbox> -TargetDatabase <2013Database>

If the Exchange Administrator does not have a mailbox, then move the Arbitration mailboxes to Exchange 2013:

Get-Mailbox –Arbitration | New-MoveRequest -TargetDatabase <2013Database>

In addition, there have been reported issues with 2013 EMS still having connectivity issues even after the relevant mailboxes have been moved. A different Windows user with appropriate Exchange permissions (using a different Windows profile) will work fine however. It seems there are PowerShell cookies for the initial profile used which could still be causing problems. In this scenario, you may have to remove all listed cookies in the following registry key (Warning, edit the registry at your own risk. A backup of the registry is recommended before making modifications):

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\WSMAN\Client\ConnectionCookies

Summary

It should be noted that while this scenario involved Exchange 2013 CU11 being installed into an existing Exchange 2010 environment, it can affect other scenarios as well:

  • Exchange 2013 CU11 or newer being installed into an existing Exchange 2010 environment
  • Exchange 2013 CU11 or newer being installed into an existing Exchange 2007 environment
  • Exchange 2016 CU1 (when released) or newer being installed into an existing Exchange 2010 environment

So unless Microsoft changes the behavior of Mailbox Anchoring, this is a precaution that should be taken when installing the first Exchange 2013 CU11/2016 CU1 (when released) server into an existing environment.

 

Edit: This forum post also describes the issue. In it, the user experiences odd behavior with the 2013 servers not being displayed if you run a Get-ExchangeServer & other odd behavior. This is similar to what I experienced in some lab testing. Ultimately, the same resolution applies.

https://social.technet.microsoft.com/Forums/en-US/05897b40-0717-437d-90ca-d550e3226c2a/exchange-2013-cu-11-breaks-some-admin-accounts-?forum=exchangesvrdeploy

 

Beware Full OAB Downloads After Installing 1st Exchange 2013 Server in Existing 07/10 Environment


Members of the Exchange 2013 Technology Adoption Program (TAP) have known about this issue for a while & the general public had the potential to figure it out once Exchange 2010 SP3 came out last month which allowed co-existence with 2013 in a lab environment; now the Exchange Team has been very clear about it with this recent blog post today. Actually, a Microsoft Support-led session at MEC was when I first heard about it in detail. So what’s the issue? Basically, you have the potential to experience an organization-wide full Offline Address Book download just as a result of installing the first Exchange 2013 server into your existing Exchange environment.

Background:

The Offline Address Book is used by Outlook Cached mode clients to be able to have offline access to Address Lists as well as some Group Metric data when they aren’t connected to the Exchange Server. For a very detailed explanation from Neil Hobson see the following article from him.

Issues can occur when the OAB for an organization grows to a large size, sometimes in the hundreds of MB. Things that contribute to this size are things like number of recipients in AD, number of distribution groups, populated user attributes, & certificate usage (reference ). It’s important to note that GAL Photos are NOT stored in the OAB. The OAB just includes a pointer to AD where the photo is actually stored (reference ). Fortunately, Exchange/Outlook is smart enough to only download the changes to the OAB instead of the entire thing every day. There are still some circumstances where the entire OAB will be downloaded again, which makes it very important to understand the size of your OAB so you know just how much of your networks bandwidth will be used when all Outlook Cached Mode clients perform a full download (reference ).

So as you might imagine, whether or not clients will perform a full OAB download becomes a topic of concern during an Exchange migration.

Offline Address Books in Exchange 2003/2007/2010 are associated with Mailbox Databases, specifically on the Properties>Client Settings Tab of the Mailbox Database in the associated Management Console:

OAB1

However, in the screenshot above you’ll notice that the “Offline Address Book” field is blank on this Mailbox Database. This is the case by default with all Mailbox Databases. This is not an issue because any Mailbox Database that has its “OfflineAddressBook” attribute set to $Null by default will use the Default Offline Address Book in the Exchange Organization. This OAB can be seen below:

OAB2

This means if you have configured multiple Offline Address Books in your environment then you would need to manually specify the additional OAB’s on the Databases you would want to use them; otherwise the default OAB would be used. Simply put, if the value for OAB is blank on a Mailbox Database then it will use the Default OAB. If it is hard-set then it will use whichever OAB you hard set it to. Some customers will hard set this value if they want Mailboxes on specific Mailbox Databases to use a specific OAB. Maybe an OAB that only contains a specific Address List instead of the entire GAL like the example below:

OAB3

Many organizations just have 1 OAB & as a result have never populated the Properties>Client Settings>Offline Address Book value of their Mailbox Databases. This is where a big issue can come into play during an Exchange 2013 migration, or even if you just want a single Exchange 2013 server in your environment for a test group of users.

Issue:

As the recent Exchange Team Blog post announcing 2013 CU1 states, you need to make sure all of your Exchange 2007/2010 Mailbox Databases have an actual value populated for their Offline Address Book. If they are currently blank, then populate them with your current Default OAB. Nothing will change in the environment as a result of this because they will continue to use their current OAB & continue to only download the OAB changes.

Failure to do this will result in each of these Mailbox Databases switching to use the Exchange 2013 Offline Address Book that gets created during installation of your first Exchange 2013 Mailbox Server. This will result in a Full OAB Download for all of your Outlook Cached Mode clients on these Mailbox Databases; a potentially nasty situation which could bring your network to its knees.

You can see the Exchange Team Blog post in question for steps on how to repoint these databases or just use the following commands which I have taken from the post:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | FT Name,OfflineAddressBook –AutoSize”

This lists all Mailbox Databases in your environment with their OfflineAddressBook attribute set to $null.

Then run:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | Set-MailboxDatabase -OfflineAddressBook (Get-OfflineAddressBook | Where {$_.IsDefault -eq $True})”

This command will grab each of these Mailbox Databases & populate the OfflineAddressBook attribute with the value of your Organizations current Default OAB. Effectively changing nothing in terms of client behavior but ensuring that when you install Exchange 2013, each of these Mailbox Databases do not switch over to using the 2013 OAB; at least not until you are ready & can stage this process, maybe one MDB at a time.

Summary:

These steps should be mandatory for any organization considering implementing an Exchange 2013 Server into their existing Exchange 2007/2010 environment.