Beware Full OAB Downloads After Installing 1st Exchange 2013 Server in Existing 07/10 Environment


Members of the Exchange 2013 Technology Adoption Program (TAP) have known about this issue for a while & the general public had the potential to figure it out once Exchange 2010 SP3 came out last month which allowed co-existence with 2013 in a lab environment; now the Exchange Team has been very clear about it with this recent blog post today. Actually, a Microsoft Support-led session at MEC was when I first heard about it in detail. So what’s the issue? Basically, you have the potential to experience an organization-wide full Offline Address Book download just as a result of installing the first Exchange 2013 server into your existing Exchange environment.

Background:

The Offline Address Book is used by Outlook Cached mode clients to be able to have offline access to Address Lists as well as some Group Metric data when they aren’t connected to the Exchange Server. For a very detailed explanation from Neil Hobson see the following article from him.

Issues can occur when the OAB for an organization grows to a large size, sometimes in the hundreds of MB. Things that contribute to this size are things like number of recipients in AD, number of distribution groups, populated user attributes, & certificate usage (reference ). It’s important to note that GAL Photos are NOT stored in the OAB. The OAB just includes a pointer to AD where the photo is actually stored (reference ). Fortunately, Exchange/Outlook is smart enough to only download the changes to the OAB instead of the entire thing every day. There are still some circumstances where the entire OAB will be downloaded again, which makes it very important to understand the size of your OAB so you know just how much of your networks bandwidth will be used when all Outlook Cached Mode clients perform a full download (reference ).

So as you might imagine, whether or not clients will perform a full OAB download becomes a topic of concern during an Exchange migration.

Offline Address Books in Exchange 2003/2007/2010 are associated with Mailbox Databases, specifically on the Properties>Client Settings Tab of the Mailbox Database in the associated Management Console:

OAB1

However, in the screenshot above you’ll notice that the “Offline Address Book” field is blank on this Mailbox Database. This is the case by default with all Mailbox Databases. This is not an issue because any Mailbox Database that has its “OfflineAddressBook” attribute set to $Null by default will use the Default Offline Address Book in the Exchange Organization. This OAB can be seen below:

OAB2

This means if you have configured multiple Offline Address Books in your environment then you would need to manually specify the additional OAB’s on the Databases you would want to use them; otherwise the default OAB would be used. Simply put, if the value for OAB is blank on a Mailbox Database then it will use the Default OAB. If it is hard-set then it will use whichever OAB you hard set it to. Some customers will hard set this value if they want Mailboxes on specific Mailbox Databases to use a specific OAB. Maybe an OAB that only contains a specific Address List instead of the entire GAL like the example below:

OAB3

Many organizations just have 1 OAB & as a result have never populated the Properties>Client Settings>Offline Address Book value of their Mailbox Databases. This is where a big issue can come into play during an Exchange 2013 migration, or even if you just want a single Exchange 2013 server in your environment for a test group of users.

Issue:

As the recent Exchange Team Blog post announcing 2013 CU1 states, you need to make sure all of your Exchange 2007/2010 Mailbox Databases have an actual value populated for their Offline Address Book. If they are currently blank, then populate them with your current Default OAB. Nothing will change in the environment as a result of this because they will continue to use their current OAB & continue to only download the OAB changes.

Failure to do this will result in each of these Mailbox Databases switching to use the Exchange 2013 Offline Address Book that gets created during installation of your first Exchange 2013 Mailbox Server. This will result in a Full OAB Download for all of your Outlook Cached Mode clients on these Mailbox Databases; a potentially nasty situation which could bring your network to its knees.

You can see the Exchange Team Blog post in question for steps on how to repoint these databases or just use the following commands which I have taken from the post:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | FT Name,OfflineAddressBook –AutoSize”

This lists all Mailbox Databases in your environment with their OfflineAddressBook attribute set to $null.

Then run:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | Set-MailboxDatabase -OfflineAddressBook (Get-OfflineAddressBook | Where {$_.IsDefault -eq $True})”

This command will grab each of these Mailbox Databases & populate the OfflineAddressBook attribute with the value of your Organizations current Default OAB. Effectively changing nothing in terms of client behavior but ensuring that when you install Exchange 2013, each of these Mailbox Databases do not switch over to using the 2013 OAB; at least not until you are ready & can stage this process, maybe one MDB at a time.

Summary:

These steps should be mandatory for any organization considering implementing an Exchange 2013 Server into their existing Exchange 2007/2010 environment.

Quick method to diagnose Exchange Active Directory Access & Service Startup Issues


Background:

My colleague Jedidiah Hammond wrote a great post awhile back on troubleshooting Exchange Service start-up issues. One of the main areas of focus of the post were issues with Active Directory Global Catalog servers. This can be considered an ad-on to that post as I’ll describe a useful method to troubleshoot Exchange permissions in Active Directory; more specifically, verifying Exchange has the proper access to the Global Catalog servers in and out of it’s respective Active Directory site.

Scenario:

Suppose you find that the Microsoft Exchange Active Directory Topology Service isn’t starting; or the System Attendant, or the Information Store service. Or perhaps the Exchange Management Console or Exchange Management Shell will not connect and is complaining of Active Directory/Global Catalog issues.
Often times this is a result of a port being blocked by Anti-V/Firewall between the Exchange Server and your Global Catalog. Or possibly a configuration issue on the network stack (IP/DNS/etc); maybe someone even powered your GC off much to your dismay. Assuming you have already worked through the above scenarios, one useful tool to verify Exchange/AD functionality is actually a very commonly used one; Event Viewer.

When you first deploy Exchange and run “setup /PrepareAD” (or you let the GUI setup do it for you) it is actually setting many of these permissions in AD. (For a list of all of these changes see this Technet article).

Steps:

Below is an excerpt from MSExchange ADAccess Informational Event ID 2080. You’ll find it occurring roughly every 15min on your Exchange Servers.
Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 1 1 7 1

This is an example of what the output should look like. You might be asking what those series of numbers represent. Well buried deep within the land of Exchange 2000 there lies a KB article explaining just that.

After reading the article you’ll find that these numbers are basically describing Exchange’s understanding of the Global Catalog servers made available to it; along with whether or not it has the proper ACLs set to be able to utilize them. If you find yourself pulling your hair out as to why Exchange is showing the symptoms I listed earlier, then look for this event on your Exchange server and you just might see something like the following:

Description:
Process STORE.EXE (PID=3376). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
 (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
Austin.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1
 Out-of-site:
Houston.ASH.ORG    CDG 1 7 7 1 0 0 1 7 1

Notice it ends with “0171” instead of “1171”. If we reference the above KB article then this tells us Exchange lacks the proper ACL’s in AD.

I’ve seen this many times with customers who have modified the Default Domain Controllers Group Policy or somehow blocked it’s use. I’ve also seen similar issues arise from unchecking “Include Inheritable Permissions from this Object’s Parent” in AD for various objects. If this is the case then please see the post I referenced earlier on how to resolve that. In addition, I’ve found re-running “setup.com /PrepareAD” to be a very useful troubleshooting step in situations such as these where you feel AD permissions may be at fault. Some customers have been weary of running this but honestly their fears stem from ignorance because “it just sounds scary” ; a quick read over the article I referenced earlier will tell you that running it again will only re-add the permissions Exchange has needed all along.
However, be aware that re-running PrepareAD may only resolve the issue temporarily as any bad Group Policies may find themselves being re-applied in about 15min so fixing the actual source of the issue should be the ultimate goal.

An additional note here is if you’re utilizing AD Split permissions with Exchange, there may be additional precautions to be taken before running PrepareAD  again.

Cant send to a moved email domain.


After removing a domain from the local Exchange 2003 server and moving to a different mail solution (cloud or on prem doesn’t matter) we were unable to send any more email to that domain at all…… it would get stuck in categorizer and return an NDR

We removed the Domain from the Email address policy as well as making sure it didn’t exist in the SMTP virtual server or any connectors.

 

After doing that we were still unable to send any email to the external domain ( mail tracking showed it getting an NDR from categorizer)

also any email sent to that domain from the new 2010 mail system on the same domain does work, so only mail from the 2003 does not work…… hmmm

After much log reading and troubleshooting the answer was in the IIS MetaBase. (increase categorizer logging and look for event 6015)

 

We had to use MetaBase explorer to remove the old removed domain from IIS\SMTP

To resolve this problem, follow these steps:

  1. Install IIS 6.0 Resource Kit Tools. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

    840671 (http://support.microsoft.com/kb/840671/ ) The IIS 6.0 Resource Kit Tools

  2. Open IIS Metabase Explorer.
  3. Expand LM, and then expand SmtpSvc.
  4. There are two items that are listed under SmtpSvc. 1 and another item If you expand the both items, you can see domain names.
  5. Right-click the invalid domain name, and then click Delete.
  6. Restart the Simple Mail Transfer Protocol service and the Microsoft Routing Engine service.

http://support.microsoft.com/kb/952841

Testing SMTP


So this is fairly basic but I realized I don’t have anything laying out the basic tools for testing SMTP posted.

Test with Telnet

  1. Determine the address of your SMTP server (unless you already know it or are testing local to the server)
    1. Open a command line (each line is a line return)
    2. type nslookup
      1. set type=mx
      2. domain.com
      3. Record the results
  2. Test with Telnet
    1. telnet <mail.domain.com> 25
      1. (this could be localhost or the output from the nslookup step)
      2. If you mistype past this point hit enter and try again don’t use backspace)
    2. ehlo mail.mydomain.com (you can replace ehlo with helo if your not using an exchange server)
    3. mail from: test@domain.com
    4. rcpt to: user@domain.com
    5. data
    6. Subject: This is a test
    7. Here you type your test message
    8. . (this is a period)
    9. quit

Web Based tools

Local Test tools

  • Pop3\SMTP mail client (outlook\express, thunderbird)
  • powershell
  • CMD\Telnet

Exchange 2007\2010 Powershell tools

  • Test-EdgeSynchronization (if using an edge subscription)
  • Test-Mailflow
  • Test-SmtpConnectivity

Supporting URLs

Cant upgrade an Address Policy after removing your 2003 exchange server.


When you try to update your address policy according to documentation  on your 2010 or 2007 Exchange:

Set-EmailAddressPolicy “Default Policy” –IncludedRecipients AllRecipients

You get the following error:
Set-EmailAddressPolicy : The recipient policy “Default Policy” with mailbox man
ager settings cannot be managed by the current version of Exchange Management C
onsole. Please use a management console with the same version as the object.
At line:1 char:23
+ Set-EmailAddressPolicy  <<<< “Default Policy” -IncludedRecipients AllRecipien

  1. Remove Mailbox manager from 2003
  2. Manually change the attrib of the Policy
    1. Start –> Run –> Adsiedit
    2. Right Click ADSI Edit –> Connect to –> Configuration
    3. image
    4. Expand Configuration Container [server_dc.yourdomain.com] –> CN=Configuration… –> CN=Services –>  CN=Microsoft Exchange –> CN=Your_Exchange_Org_Name Expand Recipient Policies
    5. image
      default policy -> properties
      MsExchPolicyOptionList value
    6. image
    7. Click Edit –> Edit
    8. image
    9. remove the MailBox Manager Policy hex Value
      • FC 1C 49 26 50 9E 57 48 86 1B 0C B8 DF 22 B5 D7 = Address List pol
      • EC 13 68 3B 89 CE BA 42 94 42 D8 7D 4A A3 0D BC = MailBox Manager Policy
  3.  

 

http://msexchangeteam.com/archive/2007/01/11/432158.aspx

Cannot install or uninstall an Exchange Role after a failed disaster recovery install


You may get an error similar to the following:

image

Solution:

  1. Delete the DR Key (if it exist)
    1. open regedit
    2. go to HKLM\Software\Microsoft\ExchangeServer\<version>\<Installed Role>
    3. Delete the DisasterRecovery Key
  2. Delete the watermarks
    1. open regedit
    2. go to HKLM\Software\Microsoft\ExchangeServer\<version>\<Installed Role>
    3. Delete the ConfiguredVersion Key
  3. Re-run the install or /m:RecoverServer

MSKB: How to Recover a Lost Exchange Server

Share

Meeting invite changes and cancelations get stuck in queue


When sending meeting changes or cancelations to another mail server outside of your exchange 2003 organization messages get stuck in the queue and you Get the following errors:

Event Type: Warning
Event Source: MSExchangeTransport
Event Category: Exchange Store Driver
Event ID: 327

If an administrator tries to open the message in the Exchange System Manager console, the administrator may receive the following error message:Unable to open for delivery

To verify this is the issue follow these steps on the message that is stuck.

  1. Launch MFCMAPI and select OK.
  2. Choose Session –> Logon –> Display Store Table
  3. Select the proflle used to open the mailbox
  4. In the returned items look for the row that has "Mailbox – <username>" and double click to open the row
  5. In the new "Mailbox – <username>" window expand the Root – Mailbox folder
  6. Expand the IPM_SUBTREE (or the mailbox) folder
  7. Open the calendar folder by double clicking on it.
  8. In the new "Calendar" window navigate to the appointment item (you can sort by Subject by clicking the Subject column)
  9. Right click the appointment item and choose "Display Recipient Table" from the menu
  10. In the recipients table scroll to the right until you can view the column named "PR_RECIPIENT_TRACKSTATUS"
  11. Note the number value for each recipient and this will indicate their tracking status on the item.
  12. If the value is 0 then it means that the tracking status is not available.

In order to fix this issue apply this hot fix.
http://support.microsoft.com/kb/938650

Error when Creating an Recovery Storage Group in 2007


You may get an error similar to this when you are trying to create a RSG in 2007

Error encountered while trying to add database (Mailbox Database) into recovery storage group (Recovery Storage Group). Error message is: The mailbox database that you specified is already associated with a recovery mailbox database. Specified mailbox database: DATA-BASE-GUID.

 

This may be caused by an existing RSG, this includes on your 2003 exchange server.

Remove any RSGs on all exchange servers and re-run the RSG creation on 2007

Recover Exchange


Have you ever had have a situation where you need to just start over with your exchange server but don’t want to lose data? (OS corruption\ Hardware Failure\ too many undocumented changes that caused an outage)

Here are the “quick and easy steps to recover”

  1. Stop the “Microsoft Exchange Information Store” service
  2. Note the OS service pack level\patches\hotfixes
  3. Copy or backup all your database files *.edb (if they are on a drive other than C you may not even have to do that)
  4. Format the C: (that’s right format it, so if you have something else on it back it up)
  5. Reinstall the OS and re-service pack it
  6. Rejoin the domain using the same server name as before (this is critical)
  7. install the exchange prerequisites.
    1. 2003
    2. 2007
    3. 2010
  8. Reinstall using exchange media
    1. 2003 = setup /disasterrecovery
    2. 2007\2010 = setup /m:recoverserver
  9. Copy or restore the exchange databases back to the original location
  10. mount the databases
  11. DONE! exchange should be back up and running!

Exchange 2007 Performance Troubleshooting


Perf Tips

  • Don’t stop on the first possible problem, continue on to be sure that’s not simple a symptom
  • Dont make any detrimental changes and ALWAYS have a backup!

If your having exchange perf issues here are some counters you should look at

  1. First the RPC Counters – these counters will show you if the clients are “feeling” a resource issue
    • MsExchangeIS\RPCAveraged Latency – should be under  50 (100 if in cached)
      • RPC Operations/Sec – Relative (Baseline\Trending
      • RPC Requests – Rec  under 70
    • If you see RPC ops go at around time of latency may be adding too much load
  2. Exchange Database health
    • MsExchangeDatabase(Information Store)\Database Page Faults Stalls/Sec  <not page faults>
      • Check health of DB it self – Page fault stalls indicate an issue writing to the DB, some are OK many are not.
      • Cache Size (Mem -2gb) – look at avail mem vs the Cache size | Check write\Read latency
      • (RTM = Database)
    • MSExchange Database\Log Record Stalls/Sec – large number = issues  <10 = Workload – this indicates an issue writing to log files
      • Correlate to disk and RPC
      • 10 MS writes recommended – solution could be add Disks, additional SG, balance servers.
      • Failure to add info into the log buffer
    • Msexchange Database\Log Threads Waiting/Sec – Disk issue
      • Correlate to \log Record Stalls/sec
      • Log Stall Issues (Disk or workload) – threads high along with log stalls indicate a workload issue, threads low indicate a disk issue
  3. Active Directory to exchange
    • Should all be Average of 50 or less, spike should not be higher than 100, all of these indicate an issue accessing a GC
      • MsexchangeADAccess\LDAP read Time (MSec)
        •  \LDAP Search
      • MSExchangeADAccessProcesses
      • MSExchangeADAccess Domain Controllers
      • MSExchangeADAccess\LDAP Reads/sec
        •  \LDAP Search/sec
  4. Hardware Counters
    • Storage – 
      • Physical or Logical Disk Read\Write Time –  Look at latency spikes in relation  to other (RPC Latency, Log Stalls, etc) – if RPC is ok disk is immaterial (unless dealing with transport or Edge)
      • Check Physical Disk or  logical if SAN or mount point
    • Memory –
      • Memory\Available Mbytes – Should always have Physical memory avail otherwise you will be paging to disk
      • Process, and Processor
        • \Working Set = RAM – See what process is using the most
        • \Virtual Bytes =- RAM + Page – See what process is using the most
        • \Private Bytes, etc  –  only it can use 486 (256 if /3gb used)
      • Note: X64 – will not crash but will start thrashing (memory leak)
    • Network
      • Network Interface\Output Queue Length – should be less than 2
        •  \Packet Outbound Errors – this is cumulative not a point in time, may have to reboot to check for new errors
        • \Current Bandwidth – correlate with NIC capability
        • Note: don’t capture loopback
    • Processor
      • Processor(_total)\% Processor Time  Average < 75%
      • Processor(_total)\% Privileged Time < half of Processor = problem, 75% real problem
      • Process(*)\% Processor – – See what process is using the most
Counters Thresholds
MSExchangeIS\RPC Averaged Latency < 25 ms
MSExchangeIS\RPC Operations/sec used a baseline:  online – .75 and 1 RPC hop, cache mode higher
MSExchangeIS\RCP Requests max 500, should be < 70
MSExchangeIS Client(*)\RPC Average Latency < 50ms on average
MSExchangeIS\RPC Client Backoff/Sec Identifies that the server is rejecting Connections
MSExchange Database\Database Page Fault Stalls/sec 0
MSExchange Database\Database Cache Size Minus 2 GB from what RAM is in System, Servers with sync – minus 3 GB
MSExchange Database\Log Record Stalls/sec Average of 10 or less, spike should not be higher than 100
MSExchange Database\Log Threads Waiting/sec Average of 10 or less
MSExchange Database(Information Store)\Log Threads Waiting Should be less than 10 on average.
MSExchangeIS Mailbox(_Total)\Messages Queued For Submission Below 50
MSExchangeADAccess*\LDAP Read Time Average of 50 or less, spike should not be higher than 100
MSExchangeADAccess*\LDAP Search Time Average of 50 or less, spike should not be higher than 100
MSExchangeADAccess*\LDAP Read/sec Average of 50 or less, spike should not be higher than 100
MSExchangeADAccess*\LDAP Search/sec Average of 50 or less, spike should not be higher than 100
Memory\Available Mbytes > 100 MB
Processor\Working Set Review baseline look for large changes
Processor\Virtual Bytes Review baseline look for large changes
Processor\Private Bytes Review baseline look for large changes
Processor(_Total)\% Processor Time Average < 75%
Processor(_Total)\% Privileged Time Remain below 75%
Processor(*)\% Processor Time Look for spikes
Network Interface\Output Queue Length\Packets Outbound Should not be > 10
Network Interface\Output Queue Length\Current Bandwidth Review baseline look for large changes
Database Drives  
LogicalDisk(*)\Avg  Disk Sec/Read below 50 MS (may need faster for +1000 users)
PhysicalDisk(*)\Avg  Disk Sec/Read below 50 MS (may need faster for +1000 users)
LogicalDisk(*)\Avg  Disk Sec/Write Below 100
PhysicalDisk(*)\Avg  Disk Sec/Write Below 100
Log Drives  
LogicalDisk(*)\Avg  Disk Sec/Read Below 20
LogicalDisk(*)\Avg  Disk Sec/Write Below 10
Temp Drives  
LogicalDisk(*)\Avg  Disk Sec/Read Below 20
LogicalDisk(*)\Avg  Disk Sec/Write Below 10
Network  
Network Interface\Output Que Length Below 2
Network Interface\Packet Outbound Errors No Greater than 0
Network Interface\current Bandwidth Match NIC capability