ESEUTIL

Issue

One of our smaller customers running Exchange 2010 SP3 UR2 was having an issue with one particular mailbox being unable to download mail items via ActiveSync on any device. The odd thing was that the folder structure would come down but no mail items would be synched. The customer said it was working fine until about a week previously.

Troubleshooting

Looking through Event Viewer in the Application logs led me to the following events from “MSExchangeIS Mailbox Store”:
10030

A mismatch was detected between a view of a folder and the actual contents of the folder. The mismatched item was ignored.

Attempts may be made to rebuild the view, but if this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.

Database: Mailbox Database

Folder: [MBX:John Smith][AllItems]

MsgHeader ID: 1110-1E6B08

Folder ID: 1110-3DA14B

View ID: 1110-3DA582

View Name: 1110-3DA14B +A-D-T301c

Document ID: 294529

Function: EcPopulateInitialMsgViewTable(Search)

Followed by:

10031

A folder view which previously experienced consistency issues has been deleted and will be rebuilt the next time it is needed.

Database: Mailbox Database

Folder: [MBX:John Smith][AllItems]

MsgHeader ID: 1110-1E6B08

Folder ID: 1110-3DA14B

View ID: 1110-3DA582

View Name: 1110-3DA14B +A-D-T301c

Function: EcAgeOutOneView

After seeing these events I came to the conclusion that there was logical corruption in this user’s Mailbox preventing ActiveSync from pulling the mail items down. So I immediately went to the handy replacement for ISINTEG, “New-MailboxRepairRequest”. (Reference1 Reference2)

So in this case I ran the following command:

New-MailboxRepairRequest -Mailbox John.Smith -CorruptionType FolderView,ProvisionedFolder,AggregateCounts,SearchFolder

The command lets you know the request was created but not much more than that. To view the logs on Mailbox Repair Requests you need to head back to the Application Log in Event Viewer (Reference )

We can see the below entries in the log:

10047

Mailbox level online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb started:

Database=Mailbox Database

Mailbox=4F1B824D-5C81-477E-B40B-418C888109F3

Flags=Detect, Fix

Tasks=SearchFolder, View, AggregateCount, ProvisionedFid

10062

Corruptions detected during online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb

Mailbox:4F1B824D-5C81-477E-B40B-418C888109F3 (John Smith)

Database:Mailbox Database

Corruption Is Fixed FID Property Resolution

“Folder View”, Yes, “1110-1E6B0C (Inbox)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0E (Sent Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B0F (Deleted Items)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B17 (Drafts)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6B1A (Tasks)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-1E6D67 (Junk E-Mail)”, 0x00000001, “Delete the corrupted view”

“Folder View”, Yes, “1110-3DA14B (AllItems)”, 0x00000001, “Delete the corrupted view”

10048

Online integrity check for request ec853fb3-1999-4911-9782-5170a31a37cb completed successfully.

We definitely found corruption so I had the user try the sync again and it worked!…….partly….. 😦

We were able to download mail items but whenever we tried replying to a message we were met with an error message. At this point I was pretty lost in terms of the logging available to me so I used an old trick, manually deleting the Device from underneath the User object in AD using ADSIEDIT (obligatory warning about using ADSIEDIT with care)

I opened ADSIEDIT (Start>Run>adsiedit.msc) & navigated to the Domain Partition (Default Naming Context). I then drilled down to the user object in question. Underneath the object you’ll find a container called CN=ExchangeActiveSyncDevices & underneath that you’ll find the various devices associated with that user.

Even if you use TestExchangeConnectivity.com’s ActiveSync test you’ll see an entry for that “device” listed here.

In my case I deleted each of these devices & had the user delete their profiles & re-create them. Unfortunately, we were getting the same errors regarding the inability to reply to messages.

Final Solution

So at this point I had but one reasonable option left, and it had been staring me in the face since I first saw the 10030 Event ID.

“Attempts may be made to rebuild the view, but if this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.”

So I created a new mailbox database as well as a move request for this mailbox & set the Bad Item Limit to 50 (since I expected further corruption that I didn’t catch before with the Repair Request). I checked the status of the move with the below command:

Get-MoveRequest | Get-MoveRequestStatistics -IncludeReport | fl

Once the command completed I was able to see that the move skipped 14 items because they were corrupted. It’s my assumption that there were other issues that were resolved from the move because a move is essentially Exchange copying all the mailbox data into an entirely new mailbox.

After this, the user was finally able to get full functionality out of their ActiveSync devices. Issue resolved!

The Full Story

After resolving the issue I was contacted for I began to ask additional questions about the environment to try & get a better idea of what could cause this type of corruption (admittedly, something I should have done from the very beginning as you’ll soon find out). I found that the customer had gone through a series of hardware issues which resulted in them ultimately running an ESEUTIL /P on their Exchange database. Upon hearing this, not only did things make a bit more sense but I realized it was time for a bit of lecturing.

Now I could spend a whole article detailing the ins & outs of ESEUTIL as well as proper database recovery practices but to be frank; ESEUTIL /P SHOULD ALWAYS BE A LAST RESORT! It is a hard recovery which essentially whacks everything out of the JET database that it doesn’t understand as valid data, in an effort to get it to mount. Ideally, if a customer’s database were in a Dirty Shutdown state & a Soft Recovery (ESEUTIL /R) failed; then the next step would be to restore the .EDB database file from backup & replay existing Transaction Logs to get the database to a current state (many Exchange backup solutions do this).

I’ve only ever had to run a /P for customers who did not have a backup & who’s only other recovery option would be manually backing up Outlook Cached mode to .PST (the ugliest of all options). Environments like these are an excellent example of customers who would be great for Office 365 because they don’t have the IT Staff to maintain a proper backup practice. Unfortunately, many individuals find themselves with a database that won’t mount & ESEUTIL /P is the first thing that turns up in their search results so they run it haphazardly. 9 out of 10 times, the database will mount & you won’t really lose much data. However, I’ve also seen a 150GB .EDB database file go down to 60GB after running a /P because an entire table or similar got whacked out of the database because it was corrupted. Bottom line, /P IS A LAST RESORT!!!!!

Back to our ActiveSync Issue. There’s one other thing that should be noted after running a /P on a database. It leaves your database in an UNSUPPORTED configuration; at least for the time being. The official word from Microsoft Support is that as soon as you run an ESEUTIL /P you should immediately run an Offline Defrag on the database (ESEUTIL /D; essentially creating a new database) & then run a New-MailboxRepairRequest on all mailboxes in it. So this really shines some light on the customer situation above. They performed a /P but performed neither of the above procedures on the database. To be honest, few customers do because the Offline Defrag is so time consuming (5-10GB/hr depending on HW) & requires downtime. This customer suffered the consequences because while their database would mount after running the /P, they still had logical corruption in the database. It just chose to adversely affect ActiveSync in this case.

Now this is where my personal practices slightly differ from those of MS Support (use at your own risk). Starting in Exchange 2010, mailbox moves are Online. So what I do is immediately create move requests for all mailboxes on the database in question to another database. The mere process of moving a mailbox should remove corruption (as seen above) & it has the benefit of allowing your users to work while the move takes place. Once the mailboxes have been moved I then run New-MailboxRepairRequest against all the mailboxes.

This isn’t always the best method, it’s just the one I use when the customer is really concerned about getting back up as soon as possible (RTO vs RPO). I’ve also seen cases where one bad database causes Store.exe to crash & bring down every other database on that server; so in that case the Offline Defrag is required. Either way, the Microsoft Support method should be your 1^st choice. The important thing is to take this as a lesson of what can happen if /P is run in ignorance. It’s not the only way logical corruption can occur but I’ve seen it as the culprit more than once.

Additional Reference:

http://www.paulhite.com/2013/05/repairing-mailbox-corruption-in.html

SUMMARY

This article describes how to use the Microsoft Exchange Server Eseutil tool (Eseutil.exe)

How to Recover a Corrupt Exchange Database

Be sure there are no hardware issues and that the File System is intact (run chkdsk /r and repeat until it doesn’t return errors), restore from backup is preferable if possible.
Add the Exchange bin to the “path” statement
1. Start -> Right click my computer -> properties -> advanced Tab -> environment variables (scroll down if you have to)
2. In the bottom pane double click “path”
3. Go to the end of the line enter a semi-colon and enter the path to the exchange bin directory
Verify that there is twice as much free space as the size of the database,
i.e if the database is 10 gb you need to have at least 20 gb free
Open command prompt
Change to the directory where the database is (usually C:\program files\exchsrv\mdbdata, if you cant find the database search for *.edb)
Type eseutil /mh <database name> (usually priv1.edb)
1. Look for the shutdown state (may have to scroll up) , if its in clean shutdown then you should be able to mount the database, but chances are if your reading this its in a dirty shutdown and you need to continue on
Type eseutil /k <database name> (usually priv1.edb) look for “Bad Checksums” if you have any, your repair possibility will be slim (just make a note of it)
Type eseutil /ml e00 we are looking for any missing logs (if none missing go on to step 9)
1. if there are any we have to move all subsequent logs. and the e00 log to another location, rename the last log to e00.log
2. Example we have logs E00021-E00035 and E00033 is missing, remove E00, E00034 and E00035
Re-name the .chk file to .oldchk
Type eseutil /r E00
1. Wait for that to complete, if it completes successfully move on to 11 otherwise run “P”
  1. P. eseutil /p <database name> this will force the database into a clean shutdown and will rip out any incomplete pages, this should only be done as a last resort
Start information store service, verify that the store is mounted in Exchange System Manager (ESM), if its not mounted right click and mount
1. Start -> All Programs -> Microsoft Exchange -> System Manager
2. Expand Administrative Group -> Administrative group -> Servers -> {Server name} -> {Storage Group}
3. Right Click Mailbox store -> mount
Dismount the store in ESM
Go back to the command window
1. type isinteg –s <servername> -fix –test alltests
2. When prompted select the store that you repaired
3. If any fixes are reported re-run until none are reported (just like chkdsk)
Type eseutil /d <database name> (usually priv1.edb)
Remount the database and should now be able to send receive mail

Command Description
Eseutil is a Jet database repair utility
Switch Action
/D Defragments the database
/R Soft recovery of the database
/G Integrity Check
/K Checksum
/P Hard Repair
/ML Log File dump
/MH Database file dump
/Y Copy
/C Restore

Isinteg is a integrity checker much like the windows chkdsk for NTFS

ESM Exchange System Manager is the GUI Management tool for exchange
Exchange 2007 changed the name to EMC Exchange Management Console