ActiveSync Synching Folders but not Mail

Issue

One of our smaller customers running Exchange 2010 SP3 UR2 was having an issue with one particular mailbox being unable to download mail items via ActiveSync on any device. The odd thing was that the folder structure would come down but no mail items would be synched. The customer said it was working fine until about a week previously.

Troubleshooting

Looking through Event Viewer in the Application logs led me to the following events from “MSExchangeIS Mailbox Store”:
10030

A mismatch was detected between a view of a folder and the actual contents of the folder. The mismatched item was ignored.

Attempts may be made to rebuild the view, but if this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.

Database: Mailbox Database

Folder: [MBX:John Smith][AllItems]

MsgHeader ID: 1110-1E6B08

Folder ID: 1110-3DA14B

View ID: 1110-3DA582

View Name: 1110-3DA14B +A-D-T301c

Document ID: 294529

Function: EcPopulateInitialMsgViewTable(Search)

Followed by:

10031

A folder view which previously experienced consistency issues has been deleted and will be rebuilt the next time it is needed.

Database: Mailbox Database

Folder: [MBX:John Smith][AllItems]

MsgHeader ID: 1110-1E6B08

Folder ID: 1110-3DA14B

View ID: 1110-3DA582

View Name: 1110-3DA14B +A-D-T301c

Function: EcAgeOutOneView

After seeing these events I came to the conclusion that there was logical corruption in this user’s Mailbox preventing ActiveSync from pulling the mail items down. So I immediately went to the handy replacement for ISINTEG, “New-MailboxRepairRequest”. (Reference1 Reference2)

So in this case I ran the following command:

New-MailboxRepairRequest -Mailbox John.Smith -CorruptionType FolderView,ProvisionedFolder,AggregateCounts,SearchFolder

The command lets you know the request was created but not much more than that. To view the logs on Mailbox Repair Requests you need to head back to the Application Log in Event Viewer (Reference )

We can see the below entries in the log:

10047

Mailbox level online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb started:

Database=Mailbox Database

Mailbox=4F1B824D-5C81-477E-B40B-418C888109F3

Flags=Detect, Fix

Tasks=SearchFolder, View, AggregateCount, ProvisionedFid

10062

Corruptions detected during online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb

Mailbox:4F1B824D-5C81-477E-B40B-418C888109F3 (John Smith)

Database:Mailbox Database

Corruption          Is Fixed FID         Property              Resolution

“Folder View”, Yes, “1110-1E6B0C (Inbox)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0C (Inbox)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B17 (Drafts)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B1A (Tasks)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6D67 (Junk E-Mail)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

10048

Online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb completed successfully.

We definitely found corruption so I had the user try the sync again and it worked!…….partly…..   😦

We were able to download mail items but whenever we tried replying to a message we were met with an error message. At this point I was pretty lost in terms of the logging available to me so I used an old trick, manually deleting the Device from underneath the User object in AD using ADSIEDIT (obligatory warning about using ADSIEDIT with care)

I opened ADSIEDIT (Start>Run>adsiedit.msc) & navigated to the Domain Partition (Default Naming Context). I then drilled down to the user object in question. Underneath the object you’ll find a container called CN=ExchangeActiveSyncDevices & underneath that you’ll find the various devices associated with that user.

Even if you use TestExchangeConnectivity.com’s ActiveSync test you’ll see an entry for that “device” listed here.

In my case I deleted each of these devices & had the user delete their profiles & re-create them. Unfortunately, we were getting the same errors regarding the inability to reply to messages.

Final Solution

So at this point I had but one reasonable option left, and it had been staring me in the face since I first saw the 10030 Event ID.

“Attempts may be made to rebuild the view, but if this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.”

So I created a new mailbox database as well as a move request for this mailbox & set the Bad Item Limit to 50 (since I expected further corruption that I didn’t catch before with the Repair Request). I checked the status of the move with the below command:

Get-MoveRequest | Get-MoveRequestStatistics -IncludeReport | fl

Once the command completed I was able to see that the move skipped 14 items because they were corrupted. It’s my assumption that there were other issues that were resolved from the move because a move is essentially Exchange copying all the mailbox data into an entirely new mailbox.

After this, the user was finally able to get full functionality out of their ActiveSync devices. Issue resolved!

The Full Story

After resolving the issue I was contacted for I began to ask additional questions about the environment to try & get a better idea of what could cause this type of corruption (admittedly, something I should have done from the very beginning as you’ll soon find out). I found that the customer had gone through a series of hardware issues which resulted in them ultimately running an ESEUTIL /P on their Exchange database. Upon hearing this, not only did things make a bit more sense but I realized it was time for a bit of lecturing.

Now I could spend a whole article detailing the ins & outs of ESEUTIL as well as proper database recovery practices but to be frank; ESEUTIL /P SHOULD ALWAYS BE A LAST RESORT! It is a hard recovery which essentially whacks everything out of the JET database that it doesn’t understand as valid data, in an effort to get it to mount. Ideally, if a customer’s database were in a Dirty Shutdown state & a Soft Recovery (ESEUTIL /R) failed; then the next step would be to restore the .EDB database file from backup & replay existing Transaction Logs to get the database to a current state (many Exchange backup solutions do this).

I’ve only ever had to run a /P for customers who did not have a backup & who’s only other recovery option would be manually backing up Outlook Cached mode to .PST (the ugliest of all options). Environments like these are an excellent example of customers who would be great for Office 365 because they don’t have the IT Staff to maintain a proper backup practice. Unfortunately, many individuals find themselves with a database that won’t mount & ESEUTIL /P is the first thing that turns up in their search results so they run it haphazardly. 9 out of 10 times, the database will mount & you won’t really lose much data. However, I’ve also seen a 150GB .EDB database file go down to 60GB after running a /P because an entire table or similar got whacked out of the database because it was corrupted. Bottom line, /P IS A LAST RESORT!!!!!

Back to our ActiveSync Issue. There’s one other thing that should be noted after running a /P on a database. It leaves your database in an UNSUPPORTED configuration; at least for the time being. The official word from Microsoft Support is that as soon as you run an ESEUTIL /P you should immediately run an Offline Defrag on the database (ESEUTIL /D; essentially creating a new database) & then run a New-MailboxRepairRequest on all mailboxes in it. So this really shines some light on the customer situation above. They performed a /P but performed neither of the above procedures on the database. To be honest, few customers do because the Offline Defrag is so time consuming (5-10GB/hr depending on HW) & requires downtime. This customer suffered the consequences because while their database would mount after running the /P, they still had logical corruption in the database. It just chose to adversely affect ActiveSync in this case.

Now this is where my personal practices slightly differ from those of MS Support (use at your own risk). Starting in Exchange 2010, mailbox moves are Online. So what I do is immediately create move requests for all mailboxes on the database in question to another database. The mere process of moving a mailbox should remove corruption (as seen above) & it has the benefit of allowing your users to work while the move takes place. Once the mailboxes have been moved I then run New-MailboxRepairRequest against all the mailboxes.

This isn’t always the best method, it’s just the one I use when the customer is really concerned about getting back up as soon as possible (RTO vs RPO). I’ve also seen cases where one bad database causes Store.exe to crash & bring down every other database on that server; so in that case the Offline Defrag is required. Either way, the Microsoft Support method should be your 1^st choice. The important thing is to take this as a lesson of what can happen if /P is run in ignorance. It’s not the only way logical corruption can occur but I’ve seen it as the culprit more than once.

 

Additional Reference:

http://www.paulhite.com/2013/05/repairing-mailbox-corruption-in.html

Uninstall Exchange Server 2013 Mailbox Role Fails With OAB Generation Server Error

Background

I recently purchased a new hard drive for server at home. It’s faster than old drives, and is perfect for my Exchange server 2013 virtual machines.  My existing environment consists of one Exchange 2013 CAS and two Exchange 2013 Mailbox servers in a DAG.  I plan to built two multirole servers on the new drive and decommission old servers gracefully.

Build/Remove

I successfully installed two multirole servers name MBX3 and MBX4, and added them into existing DAG. I removed database copies off old mailbox servers MBX1 and MBX2, and made sure they are not owning any databases. I then evicted them out of existing DAG, so Exchange 2013 binaries can be uninstalled properly.

Problem

MBX2 had no issue on uninstall process.

MBX1 was throwing a fit during pre-requisite check with following error message.

“Error: This Mailbox server is responsible for generating an Offline Address Book. Removal of Exchange Server isn’t permitted.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.CannotUninstallOABServer.aspx  “

The link takes me to a technet page that has yet to complete.

In Exchange 2010, moving OAB generation server was quite easy. All you had to do is run Move-OfflineAddressBook cmdlet, and wait for AD to replicate.  In Exchange 2013, Offline address book is now generated by a system mailbox. See blog post from Exchange Team here.

First, i need to identify OAB generation server in a DAG environment.

Step 1: Identify the mailbox database hosting organization mailbox with OAB Gen capability.

Use the following command to list the arbitration mailboxes with persisted capability of OABGen and database on which this mailbox is hosted:

Get-Mailbox -Arbitration | where {$_.PersistedCapabilities -like “*oab*”} | ft name,database

Step2: Identify the mailbox server where the database hosting organization mailbox is mounted

Use following command to identify active copy of mailbox database:

Get-MailboxDatabaseCopyStatus DatabaseName

The server that where the database is MOUNTED, is the OAB generation server. In my case, it shows the correct server MBX3.

For kicks, I created a new standalone database on one of new server, and moved OAB system mailbox over with New-Moverequest cmdlet. Move was successful, however, the uninstall failed again with same exact error.

Solution

While I can always remove MBX1 manually by deleting it from ADSIEDIT, to take the easy way out, but I decide to check one more place in AD, the properties of each Office Address Book object.  For properties of ” \Default Offline Address Book (Ex2012)”, I see this entry at bottom of the attribute list,

OffLineABServer is still pointing to MBX1. After i manually changed it to MBX3, issue was resolved.

Conclusion

Though this setting was there in Exchange 2010, but move-OfflineAddressBook cmdlet changes it to new server. This is not possible in Exchange server 2013. You can still run the command, but it returns following error,

Hopefully this will be fixed in the feature CU, that Exchange server setup ignores this setting,, and allows un-installation to succeed!

Recovering a Mailbox Database after Accidental Deletion from Active Directory

Not too long ago some colleagues wanted to know how to recover an Exchange 2010 mailbox database in the event someone was to actually delete the database object from Active Directory. Kind of an odd request since the Exchange Management Shell & the Exchange Management Console prevents you from doing this on a database with mailboxes still on it but an interesting scenario to play with.

So let’s say a hapless Jr. Admin is playing around in ADSIEDIT or maybe even LDP (Active Directory for Grown Ups if you ask Exchange Principal Program Manager Greg Taylor) & accidentally deletes a Mailbox Database object from AD. Doesn’t matter how many copies of that database you have in your DAG, its lifetime is now tied to the speed of your AD replication.

So what do you do? Likely the best answer is for you to perform an Active Directory Authoritative Restore; this assumes of course that you have one & you know how to do it. It can also mean a serious undertaking if you’re in a large global environment. I suppose you could also immediately shut down your Domain Controllers in that AD Site before it has a chance to replicate the deletion to the other AD Sites.; then setup a new DC in that site ASAP. That also has its issues in terms of outages.

So I feel that in a crappy situation you should always have a clear understanding of all your options. So here’s another one that involves the old database file switcheroo & some Exchange shell commands.

So in this scenario I have a couple test mailboxes (TestUser1 & TestUser2) on a database called TestDB1 in an Exchange 2010 Environment.

The below screenshot shows via OWA that we have an email in the inbox for TestUser1 which is ultimately residing within the .EDB Jet database file for TestDB1.

Now let’s live life on the edge & get rid of TestDB1 via ADSIEDIT (children & the squeamish may wish to divert their eyes).

Now that wasn’t that bad. Let me just run a Get-Mailbox command to verify the mailboxes are still present in Active Directory…..

Uh oh… I’m met with yellow text which states:

WARNING: The object ASH.COM\Users\TestUser1 has been corrupted, and it’s in an inconsistent state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.

It’s important to note that while the Mailbox Database object has been removed from AD, the actual database files have remained untouched; so all of the mailbox data is still intact. I didn’t test this but I imagine the database itself would still remain mounted until the next time the Information Store Service was restarted or its cache of AD info expired. However, transport to/from that database or IIS would likely be the first components to fail.

So how do we fix this using a method not already listed above? Simple, just move the existing database files to a safe location (good idea to use Eseutil /mh to confirm the database is in a clean shutdown) & then create a new Mailbox Database in EMC with the same name as the one you just accidentally deleted.

Note: We’re about to create a blank database that users could be accessing & mail can potentially flow to. So to prevent data loss or Outlook .OST files being affected, go ahead & Disable/Stop the Microsoft Exchange Transport Service as well as the RPC Client Access Service so that no mail can be delivered to this database & no Outlook clients can access it.

Once the new blank (aka Dial-Tone) database is mounted, go ahead & dismount it.

Delete all the database files within this new mailbox database. Move the original .EDB file into the new database’s location (assuming it was in a Clean Shutdown state after running the Eseutil /mh command).

Go to the Properties of the Mailbox Database in EMC & check the box for “This database can be overwritten by a restore”. You should now be able to mount the database.

Go to Exchange Management Shell & run a Clean-MailboxDatabase against this database.

You should now see the “Inconsistent/Corrupt” mailboxes showing up in the “Disconnected Mailboxes” pane in EMC (in my example below, TestUser2 isn’t showing because I already fixed him before taking this screenshot).

You can now use the “Set-Mailbox” shell command to set the Mailbox to use the Database in Active Directory. This value got nulled-out after the accidental deletion which caused the object to be corrupt/inconsistent. EX: Set-Mailbox TestUser1 –Database TestDB1

You can now set the Information Store & RPC Client Access Services back to Auto & Start them. At this point the user should be able to login to their mailbox & access their original inbox with all data still there.

And that’s it. It may not be the method your organization uses to recover from this particular situation but I feel it’s important to understand your options & if anything, this serves as a demonstration of the flexibility you have with Database Portability as well as Shell. This same process can be used in Exchange 2013 with only the Management Tools being different.