Unable to Delete Exchange 2016 Database


Edit: Microsoft has released the below KB to acknowledge this issue:

https://support.microsoft.com/en-us/kb/3093175

Edit#2: I’ve been told that this should be remedied in a future Exchange 2016 CU by having the error dialog include the -AuditLog parameter.

Symptom

After installing Exchange 2016 you may wish to delete the default Mailbox Database that is installed. Or during regular operations, you decide to remove any 2016 Mailbox Database in the environment. Of course before you do this you’ll need to move all mailboxes from the soon to be deleted database to another Mailbox Database. You could then run the following command to verify no mailboxes still exist on the database:

Get-Mailbox –Database MailboxDatabase

This should return no results

F1

However, when attempting to remove the database you receive the below message:

This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes or arbitration mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox -Database <Database ID>. To get a list of all mailbox plans in this database, run the command Get-MailboxPlan. To get a list of archive mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -Archive. To get a list of all public folder mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -PublicFolder. To get a list of all arbitration mailboxes in this database, run the command Get-Mailbox -Database <Database ID> -Arbitration. To disable a non-arbitration mailbox so that you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID>. To disable an archive mailbox so you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID> -Archive. To disable a public folder mailbox so that you can delete the mailbox database, run the command Disable-Mailbox <Mailbox ID> -PublicFolder. Arbitration mailboxes should be moved to another server; to do this, run the command New-MoveRequest <parameters>. If this is the last server in the organization, run the command Disable-Mailbox <Mailbox ID> -Arbitration -DisableLastArbitrationMailboxAllowed to disable the arbitration

F2

A common issue in Exchange 2010 and 2013 was being unfamiliar with how to view hidden Arbitration mailboxes, which are used for various system functions, and could prevent a database from being removed. To view these mailboxes you could run the following command:

Get-Mailbox –Database MailboxDatabase –Arbitration

This should return the Arbitration mailboxes installed on the first Mailbox Database in the Exchange environment. When performing a migration from a previous version of Exchange, these mailboxes should be moved to the newest version of Exchange.

F3

However, even after these mailboxes have been vacated from this database, the original error is still displayed. In Exchange 2016 RTM, the Get-Mailbox parameters which the error dialog box presents for you to try are:

  • -Archive
  • -Arbitration
  • -PublicFolder

Unfortunately, these may still display no results. There is one additional parameter, which is new to Exchange 2016, you will likely need to use:

Get-Mailbox –Database MailboxDatabase –AuditLog

This should display a system mailbox new to Exchange 2016.

F4

Once this mailbox is moved, you should be able to successfully delete the Mailbox Database.

This –AuditLog parameter seems effectively undocumented, as the TechNet Article for the Get-Mailbox command states it is reserved for internal Microsoft use. Unfortunately, we found that is not entirely the case.

 

 

ActiveSync Synching Folders but not Mail


Issue

One of our smaller customers running Exchange 2010 SP3 UR2 was having an issue with one particular mailbox being unable to download mail items via ActiveSync on any device. The odd thing was that the folder structure would come down but no mail items would be synched. The customer said it was working fine until about a week previously.

Troubleshooting

Looking through Event Viewer in the Application logs led me to the following events from “MSExchangeIS Mailbox Store”:
10030

A mismatch was detected between a view of a folder and the actual contents of the folder. The mismatched item was ignored.

Attempts may be made to rebuild the view, but if this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.

Database: Mailbox Database

Folder: [MBX:John Smith][AllItems]

MsgHeader ID: 1110-1E6B08

Folder ID: 1110-3DA14B

View ID: 1110-3DA582

View Name: 1110-3DA14B +A-D-T301c

Document ID: 294529

Function: EcPopulateInitialMsgViewTable(Search)

Followed by:

10031

A folder view which previously experienced consistency issues has been deleted and will be rebuilt the next time it is needed.

Database: Mailbox Database

Folder: [MBX:John Smith][AllItems]

MsgHeader ID: 1110-1E6B08

Folder ID: 1110-3DA14B

View ID: 1110-3DA582

View Name: 1110-3DA14B +A-D-T301c

Function: EcAgeOutOneView

After seeing these events I came to the conclusion that there was logical corruption in this user’s Mailbox preventing ActiveSync from pulling the mail items down. So I immediately went to the handy replacement for ISINTEG, “New-MailboxRepairRequest”. (Reference1 Reference2)

So in this case I ran the following command:

New-MailboxRepairRequest -Mailbox John.Smith -CorruptionType FolderView,ProvisionedFolder,AggregateCounts,SearchFolder

The command lets you know the request was created but not much more than that. To view the logs on Mailbox Repair Requests you need to head back to the Application Log in Event Viewer (Reference )

We can see the below entries in the log:

10047

Mailbox level online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb started:

Database=Mailbox Database

Mailbox=4F1B824D-5C81-477E-B40B-418C888109F3

Flags=Detect, Fix

Tasks=SearchFolder, View, AggregateCount, ProvisionedFid

10062

Corruptions detected during online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb

Mailbox:4F1B824D-5C81-477E-B40B-418C888109F3 (John Smith)

Database:Mailbox Database

Corruption          Is Fixed FID         Property              Resolution

“Folder View”, Yes, “1110-1E6B0C (Inbox)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0C (Inbox)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B17 (Drafts)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B1A (Tasks)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6D67 (Junk E-Mail)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

10048

Online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb completed successfully.

We definitely found corruption so I had the user try the sync again and it worked!…….partly…..   😦

We were able to download mail items but whenever we tried replying to a message we were met with an error message. At this point I was pretty lost in terms of the logging available to me so I used an old trick, manually deleting the Device from underneath the User object in AD using ADSIEDIT (obligatory warning about using ADSIEDIT with care)

I opened ADSIEDIT (Start>Run>adsiedit.msc) & navigated to the Domain Partition (Default Naming Context). I then drilled down to the user object in question. Underneath the object you’ll find a container called CN=ExchangeActiveSyncDevices & underneath that you’ll find the various devices associated with that user.

ADSIEDIT1

ADSIEDIT2

Even if you use TestExchangeConnectivity.com’s ActiveSync test you’ll see an entry for that “device” listed here.

In my case I deleted each of these devices & had the user delete their profiles & re-create them. Unfortunately, we were getting the same errors regarding the inability to reply to messages.

Final Solution

So at this point I had but one reasonable option left, and it had been staring me in the face since I first saw the 10030 Event ID.

“Attempts may be made to rebuild the view, but if this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.”

So I created a new mailbox database as well as a move request for this mailbox & set the Bad Item Limit to 50 (since I expected further corruption that I didn’t catch before with the Repair Request). I checked the status of the move with the below command:

Get-MoveRequest | Get-MoveRequestStatistics -IncludeReport | fl

Once the command completed I was able to see that the move skipped 14 items because they were corrupted. It’s my assumption that there were other issues that were resolved from the move because a move is essentially Exchange copying all the mailbox data into an entirely new mailbox.

After this, the user was finally able to get full functionality out of their ActiveSync devices. Issue resolved!

The Full Story

After resolving the issue I was contacted for I began to ask additional questions about the environment to try & get a better idea of what could cause this type of corruption (admittedly, something I should have done from the very beginning as you’ll soon find out). I found that the customer had gone through a series of hardware issues which resulted in them ultimately running an ESEUTIL /P on their Exchange database. Upon hearing this, not only did things make a bit more sense but I realized it was time for a bit of lecturing.

Now I could spend a whole article detailing the ins & outs of ESEUTIL as well as proper database recovery practices but to be frank; ESEUTIL /P SHOULD ALWAYS BE A LAST RESORT! It is a hard recovery which essentially whacks everything out of the JET database that it doesn’t understand as valid data, in an effort to get it to mount. Ideally, if a customer’s database were in a Dirty Shutdown state & a Soft Recovery (ESEUTIL /R) failed; then the next step would be to restore the .EDB database file from backup & replay existing Transaction Logs to get the database to a current state (many Exchange backup solutions do this).

I’ve only ever had to run a /P for customers who did not have a backup & who’s only other recovery option would be manually backing up Outlook Cached mode to .PST (the ugliest of all options). Environments like these are an excellent example of customers who would be great for Office 365 because they don’t have the IT Staff to maintain a proper backup practice. Unfortunately, many individuals find themselves with a database that won’t mount & ESEUTIL /P is the first thing that turns up in their search results so they run it haphazardly. 9 out of 10 times, the database will mount & you won’t really lose much data. However, I’ve also seen a 150GB .EDB database file go down to 60GB after running a /P because an entire table or similar got whacked out of the database because it was corrupted. Bottom line, /P IS A LAST RESORT!!!!!

Back to our ActiveSync Issue. There’s one other thing that should be noted after running a /P on a database. It leaves your database in an UNSUPPORTED configuration; at least for the time being. The official word from Microsoft Support is that as soon as you run an ESEUTIL /P you should immediately run an Offline Defrag on the database (ESEUTIL /D; essentially creating a new database) & then run a New-MailboxRepairRequest on all mailboxes in it. So this really shines some light on the customer situation above. They performed a /P but performed neither of the above procedures on the database. To be honest, few customers do because the Offline Defrag is so time consuming (5-10GB/hr depending on HW) & requires downtime. This customer suffered the consequences because while their database would mount after running the /P, they still had logical corruption in the database. It just chose to adversely affect ActiveSync in this case.

Now this is where my personal practices slightly differ from those of MS Support (use at your own risk). Starting in Exchange 2010, mailbox moves are Online. So what I do is immediately create move requests for all mailboxes on the database in question to another database. The mere process of moving a mailbox should remove corruption (as seen above) & it has the benefit of allowing your users to work while the move takes place. Once the mailboxes have been moved I then run New-MailboxRepairRequest against all the mailboxes.

This isn’t always the best method, it’s just the one I use when the customer is really concerned about getting back up as soon as possible (RTO vs RPO). I’ve also seen cases where one bad database causes Store.exe to crash & bring down every other database on that server; so in that case the Offline Defrag is required. Either way, the Microsoft Support method should be your 1st choice. The important thing is to take this as a lesson of what can happen if /P is run in ignorance. It’s not the only way logical corruption can occur but I’ve seen it as the culprit more than once.

 

Additional Reference:

http://www.paulhite.com/2013/05/repairing-mailbox-corruption-in.html