Mails Stuck In The Draft Folder


Today, I came cross another interesting mail flow issue, where all mails stuck in Draft folders for all users when they are using OWA. You can imagine that mail flow was broken, that non of users can send any mails internally or externally.

Customer has troubleshot it for over 12 hours, and has gone as far as re-install operating system and Exchange 2013 server with /RecoverServer switch, but issue remains.

When I started looking at the issue, I went through series of basic transport troubleshooting steps for Exchange 2013 multirole server, such as checking all transport related services, possible back pressure issue, and state of all server components. Of course, there is nothing wrong with them.

Running out of ideas, I checked settings of send connector, just to make sure there is nothing out of ordinary. I see this in Send Connector properties,

Image

 

There are not many reasons for any Exchange server to use External DNS server for lookups out there. For this environment, it certainly is not needed as well.

I unchecked the box, and restart transport service to speed up the process, but issue remans.

I then run get-TransportService | fl *dns*, to make sure that we don’t have any external DNS settings configured.

   Image

  Ah ha! External DNS server setting is set. I run few tests with nslookup, the DNS server did not respond to any queries. So that’s probably the reason why that mails are not flowing.

  To remove it, you have to run Set-TransportService -ExternalDNSAdapterEnabled $true -ExternalDNSServers $null.

  After restarting the transport service, all mails in the Draft folder are gone. Mail flow is restored!

Advertisements

Exchange 2010 SP3 installation fails on SBS 2011


I had an interesting issue with Exchange 2010 SP3 installation on a SBS 2011 server last night. Installation fails on the Hub Transport Server Role with following errors.

sbs 2011 upgrade sp3 error

 

This made me scratching my head. Why is it trying to remove existing certificate that is used by Exchange? It’s also the default SMTP certificate, that’s why setup was not able to remove it.

After investing further, I see this line in the PowerShell script,

Write-ExchangeSetupLog -Info “Removing default Exchange Certificate”;
Get-ExchangeCertificate | where {$_.FriendlyName.ToString() -eq “Microsoft Exchange”} | Remove-ExchangeCertificate

So it’s trying to remove default Exchange certificate that was created during the initial installation, that has friendly name “Microsoft Exchange”.

I’m thinking, there is no way the Godaddy certificate has Friendly Name “Microsoft Exchange”. After looking at the certificate properties, it is indeed the problem. The Friendly Name is showing “Microsoft Exchange”, instead of mail.domain.com.

In order for us to install SP3, we have to use SBS console to import a temporary certificate, so it updates “LeafCertThumbPrint” property in this registry key,

“HKEY_LOCAL_MACHINE\Software\Microsoft\SmallBusinessServer\Networking”

 Note: you can also update the registry manually with one of thumbprint from existing certificate that is already imported.

Exchange 2010 SP3 installs fine after the cert change.  Since we didn’t export the existing GoDaddy certificate before running SP3 setup, it was removed by the setup. In order for Exchange OA and Activesync clients  to continue function,  we have issue a new certificate request with proper Friendly Name, then import the new certificate. You can also reuse the existing certificate on GoDaddy’s website by using “Re-Key” option, but you might end up with a certificate without private key. To repair the missing private key, you can run following command
   certutil –repairstore my <serial number>

 

 

Exchange 2013 SP1 Breaks Hub Transport service


I had an issue last night that woke me up at 2 am in the morning by the On Call phone. I feel that we might see this often when Exchange admins start to applying Exchange server 2013 SP1.

After installing Exchange 2013 sp1, MSExchange Transport service hangs at “Starting”, then eventually crashes with couple of event ID’s.

“Event ID 1046, MSExchange TrasnportService

Worker process with process ID 17836 requested the service to terminate with an unhandled exception. “

“Event ID 4999, MSExchange Common

Watson report about to be sent for process id: 2984, with parameters: E12IIS, c-RTL-AMD64, 15.00.0847.032, MSExchangeTransport, M.Exchange.Net, M.E.P.WorkerProcessManager.HandleWorkerExited, M.E.ProcessManager.WorkerProcessRequestedAbnormalTerminationException, 5e2b, 15.00.0847.030. “

Only way to get Transport service to start, is to disable all receive connectors and reboot the server. Does it sound familiar? My colleague Andrew Higginbotham  wrote this article few weeks ago. Although it was a different issue, but custom receive connectors on a multirole server is the key.

In my case, this is also a multirole server(CAS and Mailbox on one box).  Hub Transport service listens on TCP port 2525, and Frontend transport listens on TCP port 25.  There are two custom receive connectors that were created with Hub Transport role. Both are listening on TCP port 25. I’m not sure why they haven’t had external mail flow issue by now, but it sure knows how to get your attention by breaking the transport service.

If we disable both custom receive connectors, transport service starts fine. So we went ahead and changed transport role from Hub Transport to Frontend Transport on both connectors with Set-ReceiveConnector powershell cmdlet, then re-enable them to test. Hub Transport service stays up without issue. Of course, we also rebooted the server to make sure that issue is fixed.

 

 

Edit. Microsoft has released the following KB addressing the issue

http://support.microsoft.com/kb/2958036

Uninstall Exchange Server 2013 Mailbox Role Fails With OAB Generation Server Error


Background

I recently purchased a new hard drive for server at home. It’s faster than old drives, and is perfect for my Exchange server 2013 virtual machines.  My existing environment consists of one Exchange 2013 CAS and two Exchange 2013 Mailbox servers in a DAG.  I plan to built two multirole servers on the new drive and decommission old servers gracefully.

Build/Remove

I successfully installed two multirole servers name MBX3 and MBX4, and added them into existing DAG. I removed database copies off old mailbox servers MBX1 and MBX2, and made sure they are not owning any databases. I then evicted them out of existing DAG, so Exchange 2013 binaries can be uninstalled properly.

Problem

MBX2 had no issue on uninstall process.

MBX1 was throwing a fit during pre-requisite check with following error message.

Error: This Mailbox server is responsible for generating an Offline Address Book. Removal of Exchange Server isn’t permitted.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.CannotUninstallOABServer.aspx  “

The link takes me to a technet page that has yet to complete.

In Exchange 2010, moving OAB generation server was quite easy. All you had to do is run Move-OfflineAddressBook cmdlet, and wait for AD to replicate.  In Exchange 2013, Offline address book is now generated by a system mailbox. See blog post from Exchange Team here.

First, i need to identify OAB generation server in a DAG environment.

Step 1: Identify the mailbox database hosting organization mailbox with OAB Gen capability.

Use the following command to list the arbitration mailboxes with persisted capability of OABGen and database on which this mailbox is hosted:

Get-Mailbox -Arbitration | where {$_.PersistedCapabilities -like “*oab*”} | ft name,database

Step2: Identify the mailbox server where the database hosting organization mailbox is mounted

Use following command to identify active copy of mailbox database:

Get-MailboxDatabaseCopyStatus DatabaseName

The server that where the database is MOUNTED, is the OAB generation server. In my case, it shows the correct server MBX3.

For kicks, I created a new standalone database on one of new server, and moved OAB system mailbox over with New-Moverequest cmdlet. Move was successful, however, the uninstall failed again with same exact error.

Solution

While I can always remove MBX1 manually by deleting it from ADSIEDIT, to take the easy way out, but I decide to check one more place in AD, the properties of each Office Address Book object.  For properties of ” \Default Offline Address Book (Ex2012)”I see this entry at bottom of the attribute list,

Image

OffLineABServer is still pointing to MBX1. After i manually changed it to MBX3, issue was resolved.

Conclusion

Though this setting was there in Exchange 2010, but move-OfflineAddressBook cmdlet changes it to new server. This is not possible in Exchange server 2013. You can still run the command, but it returns following error,

Image

Hopefully this will be fixed in the feature CU, that Exchange server setup ignores this setting,, and allows un-installation to succeed!

MCM, MCSM and MCA are no longer available from Microsoft


What a way to start your long weekend, when you received an email on early Saturday morning from Microsoft Learning, telling that your highest certification MCM/MCSM are no longer being offered!!

“We are contacting you to let you know we are making a change to the Microsoft Certified Master, Microsoft Certified Solutions Master, and Microsoft Certified Architect certifications. As technology changes so do Microsoft certifications and as such, we are continuing to evolve the Microsoft certification program. Microsoft will no longer offer Masters and Architect level training rotations and will be retiring the Masters level certification exams as of October 1, 2013. The IT industry is changing rapidly and we will continue to evaluate the certification and training needs of the industry to determine if there’s a different certification needed for the pinnacle of our program.”

Not only they have killed the all the Master certification programs, they also canceled all the scheduled classes in October and November. It stirred up outcries and displeasure from masters community all over the world. Many have blogged and twitted. Many exchanged thoughts over Exchange Ranger’s distribution list, hoping somebody from MS Learning is listening. Let’s hope we can get the messages up to top of the chain in Microsoft, bring the program back!

My thoughts on this after seeing Tim Sneath’s (Head of MSL) response from Microsoft Connect are:

1. MCM/MCSM should be kept as elite certification by going through 2-3 weeks of grueling training at Building 40, Redmond campus. It should never be as many as Tim expected out there. MSL has tried to make the tests available at SECURE testing centers without going through the class training, which significantly reduced the costs, and made it easier for international community to certify. Downside of that, it watered down difficulty level of knowledge exams. Questions are now from trivia facts from Microsoft Technet, instead of materials we went through the rotation. At least the lab exam is still very difficult to pass without master level knowledge on the product. I really think we should go back to old MCM model from beginning, that training class is required. When Product Group started ranger/MCM programs, it was never about how much money can the classes and certifications make. It was about having the best technical person out there for customers, increasing customer’s faith on the product, and reducing high level escalations to product group.

2. “It’s cloud or bust”. That has been the model for Microsoft since they released office 365 and Windows Azure. For example, to enable automatic site fail over for Exchange server 2013 DAG, the 3rd AD site is required for file share witness. Guess what, Exchange will soon be supporting to have that 3rd AD site in Azure. I don’t agree the direction that we are heading in the IT industry. On-premises are not going away anytime soon. Existing Lync/Exchange/Sharepoint MCM/MCSM’s are the main resources on blog sites and customer facing jobs on implementing Hybrid solutions with cloud. Microsoft’s Office 365 support has not improved since they launched back in 2011. With wave 15 office 365, more bugs and issues are produced. I already know few large customers changed their minds on migrating to office 365 because of the bugs they encountered during migration. Sure, Microsoft eventually will fix all the issues and bugs, but trusts have already broken.

Bottom line, MCM/MCSM programs should be brought back, and keep the requirement for training class. They should also stop using Technet trivia for knowledge exam questions. Questions should be written from the instructors, from the materials that was covered in the class. Retake can still happen at Prometric secure test centers. Less likely, anybody will try to brain dump the questions because so much money was paid for the rotation.

How to use Eseutil.exe to perform actions while databases are online


Some new tricks I learned today on good old Esetuil.exe tool in Exchange 2010 SP3 and Exchange 2013.

We all know, when database is online and mounted, you won’t be able to perform any actions with Eseutil.exe.
For example,

Get-MailboxDatabaseCopyStatus returns

Database e15db3 is mounted on Exchange 2013 server mbx1.

If you try to run eseutil /mh or eseutil /y, you would receive following error,

Now, with new Eseutil switches introduced in Exchange 2010 sp3 and Exchange 2013, you can perform actions while database is online and mounted.

/vss switches utilize Windows VSS engine and snapshot to perform the tasks that you traditionally have to dismount database first.

If you run eseutil /mh /vss, it will dump database info with “Dirty Shutdown” status, because it did not play the missing logs into the snapshot.
So, you want to run eseutil /mh /vss /vssrec eNN “logpath” for optimal result.

 

If you want to perform quick database backup, you can now run eseutil /y /d /vss /vssrec to achieve that, without using any type of backup software. This is my favorite, and the most useful action!

 

You can verify the backup file with eseutil /mh

 

You now, have an up to date database backup!

Exchange 2010 EMC cannot access AD configuration data after you demote a DC


Environment:

Exchange 2010/Domain Controller combo server running on Windows 2008 R2.

Problem:

Demote Domain Controller role, causes Exchange Management Console fails to retrieve any Exchange information with error message “Active directory response: The LDAP server is unavailable.”  It’s still looking for the demoted DC although it’s been cleaned out of AD/DNS. All Exchange services start fine, and Exchange Shell works fine.

Reason:

     The obsolete information is cached in an Exchange Management Console file in the Windows profile for the user. EMC is trying to connect to orginal DC that is stored in the file.

Solution:

   Go to the following folder and delete the Exchange Management Console file.

   C:\users\<specific user>\AppData\Roaming\Microsoft\MMC\Exchange Management Console

   Close EMC and reopen it.

Reference: http://support.microsoft.com/kb/2019500