Unable to open Local Windows Backup Snap-in.

Background:

• I had installed the beta version of Microsoft online backup
• I had backed up locally to a USB drive that has since failed
• Backups were scheduled locally and to the online backup service.
• When I removed the online beta backup software (Now Azure) and my failed drive, I was no longer able to manage windows backup from the GUI.

This is the error I received in the event log.

Event ID 1000
Source Application Error

Faulting application name: wbengine.exe, version: 6.2.9200.16384, time stamp: 0x50108cb6
Faulting module name: wbengine.exe, version: 6.2.9200.16384, time stamp: 0x50108cb6
Exception code: 0xc0000005
Fault offset: 0x000000000012623a
Faulting process id: 0×2678
Faulting application start time: 0x01ce64c42da7256f
Faulting application path: C:\Windows\system32\wbengine.exe
Faulting module path: C:\Windows\system32\wbengine.exe
Report Id: 6c2d3105-d0b7-11e2-9415-c86000003091
Faulting package full name:
Faulting package-relative application ID:

Cause:

I had backups placed on a failed drive, this was causing the backup software to crash when it tried to enumerate them. (Not that the error or events point to that at all!)

Resolution:

I ran the following PowerShell cmtlets and re-setup my backups (Caution this will remove all record of any backup have taken place!!)

1. Get-WBPolicy | Remove-WBPolicy
2. Remove-WBBackupSet
3. Remove-WBCatalog
4. get-Service *wb* | Start-Service
5. Restart Windows Server Backup

Sweet! may backup works again!

Note: I was also able to re-download the Azure Backup agent and that is now working like a charm as well.

Quick Exchange 2013 DAG Setup Guide

Background:

Had a co-worker ask for some basic DAG setup instructions in Exchange 2013 so I wrote a quick little guide. This covers the high points around creating the DAG as well as configuring the DAG member NICs & networks.

Step 1 – Pre-Stage DAG Computer Account
Reference. When deploying a DAG on Exchange Servers running Server 2012 you need to pre-stage the DAG computer account. The above link points to the official TechNet article for doing this but here are the basics of it:

• Create a Computer Account in AD with the name of the DAG. For example, DAG-A.
• Disable the Computer Account.
• In Active Directory Users & Computers click View>Advanced Features. Go to the Computer Account & select Properties>Security tab.
• From here you have two options; either Grant the Exchange Trusted Subsystem Full Control permissions to the DAG Computer Account or give the Computer Account of the first node you plan to join to the DAG Full Control permissions over the DAG Computer Account Object.
• Reference2

Step 2 – Configure DAG NIC’s
Reference. Exchange 2013 performs automatic DAG network configuration depending on how the NIC’s are configured. This means if the NIC’s are configured correctly then you should not have to manually collapse the DAG Networks post DAG Setup. Upon adding the nodes to the DAG, it looks for the following properties on the NICs & makes a decision based on them:

• NIC Binding Order
• Default Gateway Present
• Register DNS Checked

The DAG needs to separate MAPI/Public networks from Replication networks. This enables the DAG to properly utilize a network that the administrator has provisioned for Replication traffic & to only use the MAPI/Public networks for Replication if the Replication networks are down.

You want your MAPI/Public NICs to be top of the binding order in the OS & any Replication, Management, Backup, or iSCSI networks at the bottom of the binding order. This is a Core Windows Networking best practice as well as what the DAG looks for when trying to determine which NIC’s will be associated with the MAPI/Public DAG Networks.

The DAG also looks for the presence of a Default Gateway on the MAPI/DAG network NIC. Going along with another Windows Networking best practice, you should only have 1 Default Gateway configured in a Windows OS. If you have additional networks with different subnets on the DAG nodes then you would need to add static routes on each of the nodes using NETSH. More on this later.

Finally, NIC Properties>IPv4 Properties>Advanced>DNS>Register this connection’s addresses in DNS should be unchecked on all adapters except for the MAPI/Public NICs. This means all Replication, iSCSI, dedicated backup or management NICs should have this option unchecked. Again, this is a Windows Networking best practice but is vital for proper Automatic DAG Network Configuration in Exchange 2013.

Step 3 – Configure Routing if Needed (optional depending on DAG design)
If your DAG stretches subnets & you’re using dedicated Replication networks then they should be on their own subnet isolated from the MAPI/Public network. A common setup for a network such as this might be:

Site-Austin:
MAPI Network 192.168.1.0/24; Default Gateway 192.168.1.254
Replication Network 10.0.1.0/24; Default Gateway $Null

Site-Houston:
MAPI Network 192.168.2.0/24; Default Gateway 192.168.2.254
Replication Network 10.0.2.0/24; Default Gateway $Null

Now with the above configuration you would have some form of routing taking place between the two MAPI subnets. You would also have routing between the two Replication subnets. However, because you should only have 1 Default Gateway configured per server, DAG nodes in each site would be unable to communicate with each other over the Replication networks. This is where static routes come into play. You would run the following commands on the nodes to allow them to ping across to each other between the 10.0.1.x & 10.0.2.x networks (in the below example, REPL is the name of each node’s Replication NIC):

On Nodes in Site-Austin: “netsh interface ip add route 10.0.2.0/24 “REPL” 0.0.0.0”

On Nodes in Site-Houston: “netsh interface ip add route 10.0.1.0/24 “REPL” 0.0.0.0”

This is the preferred format for this command. There are some references to using the local interface IP instead of 0.0.0.0 but the format I use above is what is recommended by the Windows Networking Team. Reference.

“According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.”

This all assumes there is physical routing in place between the two subnets, like a Router, layer 3 Switch, or a shared virtual network in Hyper-V/ESX.

Verify connectivity between nodes over these 10.0.x.x networks using Tracert or Pathping. Note that these steps are only required if your DAG spans subnets & has replication networks in different subnets. While it technically should work, it is not recommended to stretch subnets for DAG Networks across the WAN.

It should also be noted that there should be no routing between the MAPI Networks & the Replication Networks. They should be on isolated networks that have no contact with each other. Also, Microsoft wants no greater than 500ms round trip latency between DAG nodes when you have DAG members across latent network connections. It’s important for customers to realize that you should not set your expectations around this number alone. You could easily have a connection over 500ms & not experience copy queues if you have only 20 mailboxes with low usage profiles. Alternatively, you could have a connection with only 50ms of round-trip latency but see high copy queues if you have thousands of high-usage mailboxes & a small bandwidth pipe. Just know that this number is not an end all be all.

Step 4 – Create DAG & Add Nodes
This part is pretty straightforward & you can use the EAC to do it. Just remember to give the DAG an IP address in every MAPI subnet where you have DAG nodes. So in our scenario above you would give the DAG 2 IP addresses; one in the 192.168.1.0 subnet & another in the 192.168.2.0 subnet.

Step 5 – Manually configure DAG Networks if needed
Reference. If you have dedicated management networks, dedicated backup networks, or iSCSI NIC’s then you would actually have to perform some manual steps after your DAG is setup. These networks should be ignored by the DAG & for cluster use. In order to do this we must first enable Manual DAG Network Configuration, which is disabled by default. We would then need to configure the iSCSI or similar network to be ignored by the cluster. Perform the following steps:

• Get-DatabaseAvailabilityGroup
• Set-DatabaseAvailabilityGroup <DAGName> -ManualDagNetworkConfiguration:$True
• Get-DatabaseAvailabilityGroupNetwork
• Set-DatabaseAvailabilityGroupNetwork <iSCSI/Backup/Mgmt NetworkName> -IgnoreNetwork:$True

Finally, let’s validate everything. Run the below command:

Get-DatabaseAvailabilityGroupNetwork | Format-List Identity,ReplicationEnabled,IgnoreNetwork

Verify that the iSCSI/Backup/Mgmt networks have IgnoreNetwork set to True (the MAPI & Replication networks should have this set to False). Also verify that the Replication Networks have ReplicationEnabled set to True. Finally, verify that the MAPI network has ReplicationEnabled set to False. This prevents the MAPI network from being used for Replication by default. It can still be used for Replication if all other possible replication paths go down.

References:
http://technet.microsoft.com/en-us/library/ff367878.aspx

http://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx

http://blogs.technet.com/b/scottschnoll/archive/2012/10/01/storage-high-availability-and-site-resilience-in-exchange-server-2013-part-2.aspx

http://blogs.technet.com/b/askcore/archive/2009/05/26/active-route-gets-removed-on-windows-2008-failover-cluster-ip-address-offline.aspx

http://technet.microsoft.com/en-us/library/dd298008(v=exchg.141).aspx

Creating Custom DLP Classification Rules and Policy

When at first I was looking into this the TechNet documentation was extensive and yet not as specific as I would prefer, so here is the quick and dirty DLP classification!

Creating and importing custom Classifications

1. First you need to create your custom policy XML (Example Below)
2. Save as XML Unicode file type (C:\MyNewPolicy.xml)
3. Open the XML in internet explorer if its formatted correctly you will see the XML.
4. Then import with Powershell
   New-ClassificationRuleCollection –FileData ([Byte[]]$(Get-Content -path C:\MyNewPolicy.xml -Encoding byte -ReadCount 0))
5. Once its imported you should be able to create a new DLP policy using the EAC

Creating a custom DLP Rule

1. Login to EAC (i.e https://mail.domain.com/ecp)
2. Click Compliance Management, data loss prevention
3. Click the Plus , then New custom policy
   
4. Name your policy and Choose your mode (I like to test with Policy tags), and click Save
   
5. Select the policy and click the edit your new policy
6. Select Rules from the left
7. Click the to Create a new rule
8. On the Apply this rule if field choose The message contains Sensitive information..
9. Click *Select sensitive information types….. (if applicable)
10. Click the to choose from the list,
11. You should now see your new classification (from the example below it would be Secure Product Codes\ DLP by Exchangemasters.info)

Useful Tools

• Regex – http://gskinner.com/RegExr/
• GUID creator – http://www.guidgenerator.com/online-guid-generator.aspx
• Technet – http://technet.microsoft.com/en-us/library/jj674704(v=exchg.150).aspx

 

Example of a Rule Classification XML

<?xml version=”1.0″ encoding=”utf-16″?>

<RulePackage xmlns=”http://schemas.microsoft.com/office/2011/mce”&gt;

 

  <RulePack id=”b4b4c60e-2ff7-47b2-a672-86e36cf608be”>

    <Version major=”1″ minor=”0″ build=”0″ revision=”0″/>

    <Publisher id=”7ea13c35-0e58-472a-b864-5f2e717edec6″/>

    <Details defaultLangCode=”en-us”>

      <LocalizedDetails langcode=”en-us”>

        <PublisherName>DLP by Exchangemasters.info</PublisherName>

        <Name>Secure Product Codes</Name>

        <Description>Secure Products</Description>

      </LocalizedDetails>

    </Details>

  </RulePack>

 

  <Rules>

 

    <!– Product Code –>

    <Entity id=”acc59528-ff01-433e-aeee-13ca8aaee159″ patternsProximity=”300″ recommendedConfidence=”75″>

      <Pattern confidenceLevel=”75″>

        <IdMatch idRef=”Regex_Product_Code” />

        <Match idRef=”Code” />

      </Pattern>

    </Entity>

 

    <Regex id=”Regex_Product_Code”>[A-Z]{3}[0-9]{9}

</Regex>

    <Keyword id=”Code”>

      <Group matchStyle=”word”>

        <Term>Code</Term>

      </Group>

    </Keyword>

 

 

    <LocalizedStrings>

 

      <Resource idRef=”acc59528-ff01-433e-aeee-13ca8aaee159″>

        <Name default=”true” langcode=”en-us”>

          Product Code

        </Name>

        <Description default=”true” langcode=”en-us”>

          A custom classification for detecting product codes that have 3 uppercase letters and 9 numbers

        </Description>

      </Resource>

 

    </LocalizedStrings>

  </Rules>

</RulePackage>

New behavior in Outlook 2013 causing certificate errors in some environments

Background:

I originally discovered this issue back in early Feb & let a couple people on the Exchange Product Team know about it via the TAP but it seems to be affecting more customers than initially thought so I thought I’d share.

In Outlook 2007 through Outlook 2010 all domain-joined Outlook clients would initially query Active Directory for AutoDiscover information & ultimately find a Service Connection Point (SCP) value that would point them to their nearest Client Access Server’s AutoDiscover virtual directory. If that failed then they would revert to using DNS like any non-domain-joined Outlook client. The order of this non-domain-joined lookup is as follows:

https://company.com/autodiscover/autodiscover.xml

https://autodiscover.company.com/autodiscover/autodiscover.xml

Local XML File

http://company.com/autodiscover/autodiscover.xml (looking for a redirect website)

SCP AutoDiscover Record

Why it ever looked to https://company.com/autodiscover/autodiscover.xml I’ll never really know because honestly I’ve never come across a customer who had it deployed that way; most have https://autodiscover.company.com/autodiscover/autodiscover.xml but I imagine when Exchange 2007 was first being developed they weren’t exactly sure how customers would be implementing AutoDiscover.

Issue:

The above methods have served us well since Exchange 2007 timeframe but for some reason the Outlook team decided to try & implement some giddyup into Outlook & try to speed up the process. They decided to have domain-joined Outlook 2013 clients query both the SCP values in AD as well as the DNS records at the same time. If an SCP record was found it would still be used but in the event it failed then it would already have the DNS response ready to go. Great idea, however there’s one problem in the implementation.

If Outlook 2013 encounters any kind of Certificate error while doing the simultaneous DNS query then you will receive a pop-up in Outlook about the cert.

I actually stumbled upon this while in the middle of the scenario below:

That’s right, I actually get a certificate pop-up for my lab’s domain name (ash15.com) & not autodiscover.ash15.com like one would expect if I were to have a certificate issue on Exchange.

When Outlook 2013 does it’s simultaneous DNS AutoDiscover query the first URL it tries is https://company.com/autodiscover/autodiscover.xml, which in my lab environment resolved to my Domain Controller, which was also serving DNS, as well as a Certificate Authority. Ash15.com resolved to this server because it’s my internal Active Directory domain name & the name server entry resolves to my DC (just ping internaldomainname.local in your AD lab environment & you’ll see the same thing).

Now because I have web enrollment enabled & am listening on 443 in IIS the server responded. Also, because I did not have a cert installed on the server with ash15.com in the Subject or Subject Alternative Name then it gave the certificate error we see above.

Resolution:

The error is easy enough to get through & it only occurred on initial profile creation but this can definitely prove painful for some customers. Obviously my lab environment is a corner case but there have been several other customers report this issue with Outlook 2013 as well.

Here’s an example scenario.

Imagine you have a public website for andrewswidgets.com hosted by a third-party hosting site & you did not pay for HTTPS/443 services. However if you were to query the website using https then it could respond & obviously not return a certificate with andrewswidgets.com on it (because you haven’t paid for it you cheapskate…). Now imagine you begin deploying users using Outlook 2013 in your internal environment. In the past, they would have found the SCP record that would have pointed them to your internal Exchange 07/10/13 server for AutoDiscover & would have been happy as a clam (one Exchange Product Manager’s favorite way to describe Exchange bliss). However, now they may get a certificate pop-up for andrewswidgets.com when creating a new profile.

There are a couple ways around this. Make sure andrewswidgets.com doesn’t listen on 443, or possibly get a proper cert on your website that is listening on 443. Simply put, just make sure whatever andrewswidgets.com resolves to is something that’s not going to throw a certificate error.

I’ve heard nothing concrete or public but the Outlook team is aware of the issue & listening to customer feedback. I suggest contacting Microsoft Support if your organization is running into this issue.

 

Also, this KB offers methods to control which AutoDiscover methods are used by your Outlook clients

 

How to use Eseutil.exe to perform actions while databases are online

Some new tricks I learned today on good old Esetuil.exe tool in Exchange 2010 SP3 and Exchange 2013.

We all know, when database is online and mounted, you won’t be able to perform any actions with Eseutil.exe.
For example,

Get-MailboxDatabaseCopyStatus returns

Database e15db3 is mounted on Exchange 2013 server mbx1.

If you try to run eseutil /mh or eseutil /y, you would receive following error,

Now, with new Eseutil switches introduced in Exchange 2010 sp3 and Exchange 2013, you can perform actions while database is online and mounted.

/vss switches utilize Windows VSS engine and snapshot to perform the tasks that you traditionally have to dismount database first.

If you run eseutil /mh /vss, it will dump database info with “Dirty Shutdown” status, because it did not play the missing logs into the snapshot.
So, you want to run eseutil /mh /vss /vssrec eNN “logpath” for optimal result.

 

If you want to perform quick database backup, you can now run eseutil /y /d /vss /vssrec to achieve that, without using any type of backup software. This is my favorite, and the most useful action!

 

You can verify the backup file with eseutil /mh

 

You now, have an up to date database backup!

Its official

Well its official the next Microsoft Exchange Conference is going to be in our own little town of Austin TX in 2014!

http://www.iammec.com/

Beware Full OAB Downloads After Installing 1st Exchange 2013 Server in Existing 07/10 Environment

Members of the Exchange 2013 Technology Adoption Program (TAP) have known about this issue for a while & the general public had the potential to figure it out once Exchange 2010 SP3 came out last month which allowed co-existence with 2013 in a lab environment; now the Exchange Team has been very clear about it with this recent blog post today. Actually, a Microsoft Support-led session at MEC was when I first heard about it in detail. So what’s the issue? Basically, you have the potential to experience an organization-wide full Offline Address Book download just as a result of installing the first Exchange 2013 server into your existing Exchange environment.

Background:

The Offline Address Book is used by Outlook Cached mode clients to be able to have offline access to Address Lists as well as some Group Metric data when they aren’t connected to the Exchange Server. For a very detailed explanation from Neil Hobson see the following article from him.

Issues can occur when the OAB for an organization grows to a large size, sometimes in the hundreds of MB. Things that contribute to this size are things like number of recipients in AD, number of distribution groups, populated user attributes, & certificate usage (reference ). It’s important to note that GAL Photos are NOT stored in the OAB. The OAB just includes a pointer to AD where the photo is actually stored (reference ). Fortunately, Exchange/Outlook is smart enough to only download the changes to the OAB instead of the entire thing every day. There are still some circumstances where the entire OAB will be downloaded again, which makes it very important to understand the size of your OAB so you know just how much of your networks bandwidth will be used when all Outlook Cached Mode clients perform a full download (reference ).

So as you might imagine, whether or not clients will perform a full OAB download becomes a topic of concern during an Exchange migration.

Offline Address Books in Exchange 2003/2007/2010 are associated with Mailbox Databases, specifically on the Properties>Client Settings Tab of the Mailbox Database in the associated Management Console:

However, in the screenshot above you’ll notice that the “Offline Address Book” field is blank on this Mailbox Database. This is the case by default with all Mailbox Databases. This is not an issue because any Mailbox Database that has its “OfflineAddressBook” attribute set to $Null by default will use the Default Offline Address Book in the Exchange Organization. This OAB can be seen below:

This means if you have configured multiple Offline Address Books in your environment then you would need to manually specify the additional OAB’s on the Databases you would want to use them; otherwise the default OAB would be used. Simply put, if the value for OAB is blank on a Mailbox Database then it will use the Default OAB. If it is hard-set then it will use whichever OAB you hard set it to. Some customers will hard set this value if they want Mailboxes on specific Mailbox Databases to use a specific OAB. Maybe an OAB that only contains a specific Address List instead of the entire GAL like the example below:

Many organizations just have 1 OAB & as a result have never populated the Properties>Client Settings>Offline Address Book value of their Mailbox Databases. This is where a big issue can come into play during an Exchange 2013 migration, or even if you just want a single Exchange 2013 server in your environment for a test group of users.

Issue:

As the recent Exchange Team Blog post announcing 2013 CU1 states, you need to make sure all of your Exchange 2007/2010 Mailbox Databases have an actual value populated for their Offline Address Book. If they are currently blank, then populate them with your current Default OAB. Nothing will change in the environment as a result of this because they will continue to use their current OAB & continue to only download the OAB changes.

Failure to do this will result in each of these Mailbox Databases switching to use the Exchange 2013 Offline Address Book that gets created during installation of your first Exchange 2013 Mailbox Server. This will result in a Full OAB Download for all of your Outlook Cached Mode clients on these Mailbox Databases; a potentially nasty situation which could bring your network to its knees.

You can see the Exchange Team Blog post in question for steps on how to repoint these databases or just use the following commands which I have taken from the post:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | FT Name,OfflineAddressBook –AutoSize”

This lists all Mailbox Databases in your environment with their OfflineAddressBook attribute set to $null.

Then run:

“Get-MailboxDatabase | Where {$_.OfflineAddressBook -eq $Null} | Set-MailboxDatabase -OfflineAddressBook (Get-OfflineAddressBook | Where {$_.IsDefault -eq $True})”

This command will grab each of these Mailbox Databases & populate the OfflineAddressBook attribute with the value of your Organizations current Default OAB. Effectively changing nothing in terms of client behavior but ensuring that when you install Exchange 2013, each of these Mailbox Databases do not switch over to using the 2013 OAB; at least not until you are ready & can stage this process, maybe one MDB at a time.

Summary:

These steps should be mandatory for any organization considering implementing an Exchange 2013 Server into their existing Exchange 2007/2010 environment.