Unable to open Local Windows Backup Snap-in.

Background:

• I had installed the beta version of Microsoft online backup
• I had backed up locally to a USB drive that has since failed
• Backups were scheduled locally and to the online backup service.
• When I removed the online beta backup software (Now Azure) and my failed drive, I was no longer able to manage windows backup from the GUI.

This is the error I received in the event log.

Event ID 1000
Source Application Error

Faulting application name: wbengine.exe, version: 6.2.9200.16384, time stamp: 0x50108cb6
Faulting module name: wbengine.exe, version: 6.2.9200.16384, time stamp: 0x50108cb6
Exception code: 0xc0000005
Fault offset: 0x000000000012623a
Faulting process id: 0×2678
Faulting application start time: 0x01ce64c42da7256f
Faulting application path: C:\Windows\system32\wbengine.exe
Faulting module path: C:\Windows\system32\wbengine.exe
Report Id: 6c2d3105-d0b7-11e2-9415-c86000003091
Faulting package full name:
Faulting package-relative application ID:

Cause:

I had backups placed on a failed drive, this was causing the backup software to crash when it tried to enumerate them. (Not that the error or events point to that at all!)

Resolution:

I ran the following PowerShell cmtlets and re-setup my backups (Caution this will remove all record of any backup have taken place!!)

1. Get-WBPolicy | Remove-WBPolicy
2. Remove-WBBackupSet
3. Remove-WBCatalog
4. get-Service *wb* | Start-Service
5. Restart Windows Server Backup

Sweet! may backup works again!

Note: I was also able to re-download the Azure Backup agent and that is now working like a charm as well.

Quick Exchange 2013 DAG Setup Guide

Background:

Had a co-worker ask for some basic DAG setup instructions in Exchange 2013 so I wrote a quick little guide. This covers the high points around creating the DAG as well as configuring the DAG member NICs & networks.

Step 1 – Pre-Stage DAG Computer Account
Reference. When deploying a DAG on Exchange Servers running Server 2012 you need to pre-stage the DAG computer account. The above link points to the official TechNet article for doing this but here are the basics of it:

• Create a Computer Account in AD with the name of the DAG. For example, DAG-A.
• Disable the Computer Account.
• In Active Directory Users & Computers click View>Advanced Features. Go to the Computer Account & select Properties>Security tab.
• From here you have two options; either Grant the Exchange Trusted Subsystem Full Control permissions to the DAG Computer Account or give the Computer Account of the first node you plan to join to the DAG Full Control permissions over the DAG Computer Account Object.
• Reference2

Step 2 – Configure DAG NIC’s
Reference. Exchange 2013 performs automatic DAG network configuration depending on how the NIC’s are configured. This means if the NIC’s are configured correctly then you should not have to manually collapse the DAG Networks post DAG Setup. Upon adding the nodes to the DAG, it looks for the following properties on the NICs & makes a decision based on them:

• NIC Binding Order
• Default Gateway Present
• Register DNS Checked

The DAG needs to separate MAPI/Public networks from Replication networks. This enables the DAG to properly utilize a network that the administrator has provisioned for Replication traffic & to only use the MAPI/Public networks for Replication if the Replication networks are down.

You want your MAPI/Public NICs to be top of the binding order in the OS & any Replication, Management, Backup, or iSCSI networks at the bottom of the binding order. This is a Core Windows Networking best practice as well as what the DAG looks for when trying to determine which NIC’s will be associated with the MAPI/Public DAG Networks.

The DAG also looks for the presence of a Default Gateway on the MAPI/DAG network NIC. Going along with another Windows Networking best practice, you should only have 1 Default Gateway configured in a Windows OS. If you have additional networks with different subnets on the DAG nodes then you would need to add static routes on each of the nodes using NETSH. More on this later.

Finally, NIC Properties>IPv4 Properties>Advanced>DNS>Register this connection’s addresses in DNS should be unchecked on all adapters except for the MAPI/Public NICs. This means all Replication, iSCSI, dedicated backup or management NICs should have this option unchecked. Again, this is a Windows Networking best practice but is vital for proper Automatic DAG Network Configuration in Exchange 2013.

Step 3 – Configure Routing if Needed (optional depending on DAG design)
If your DAG stretches subnets & you’re using dedicated Replication networks then they should be on their own subnet isolated from the MAPI/Public network. A common setup for a network such as this might be:

Site-Austin:
MAPI Network 192.168.1.0/24; Default Gateway 192.168.1.254
Replication Network 10.0.1.0/24; Default Gateway $Null

Site-Houston:
MAPI Network 192.168.2.0/24; Default Gateway 192.168.2.254
Replication Network 10.0.2.0/24; Default Gateway $Null

Now with the above configuration you would have some form of routing taking place between the two MAPI subnets. You would also have routing between the two Replication subnets. However, because you should only have 1 Default Gateway configured per server, DAG nodes in each site would be unable to communicate with each other over the Replication networks. This is where static routes come into play. You would run the following commands on the nodes to allow them to ping across to each other between the 10.0.1.x & 10.0.2.x networks (in the below example, REPL is the name of each node’s Replication NIC):

On Nodes in Site-Austin: “netsh interface ip add route 10.0.2.0/24 “REPL” 0.0.0.0”

On Nodes in Site-Houston: “netsh interface ip add route 10.0.1.0/24 “REPL” 0.0.0.0”

This is the preferred format for this command. There are some references to using the local interface IP instead of 0.0.0.0 but the format I use above is what is recommended by the Windows Networking Team. Reference.

“According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.”

This all assumes there is physical routing in place between the two subnets, like a Router, layer 3 Switch, or a shared virtual network in Hyper-V/ESX.

Verify connectivity between nodes over these 10.0.x.x networks using Tracert or Pathping. Note that these steps are only required if your DAG spans subnets & has replication networks in different subnets. While it technically should work, it is not recommended to stretch subnets for DAG Networks across the WAN.

It should also be noted that there should be no routing between the MAPI Networks & the Replication Networks. They should be on isolated networks that have no contact with each other. Also, Microsoft wants no greater than 500ms round trip latency between DAG nodes when you have DAG members across latent network connections. It’s important for customers to realize that you should not set your expectations around this number alone. You could easily have a connection over 500ms & not experience copy queues if you have only 20 mailboxes with low usage profiles. Alternatively, you could have a connection with only 50ms of round-trip latency but see high copy queues if you have thousands of high-usage mailboxes & a small bandwidth pipe. Just know that this number is not an end all be all.

Step 4 – Create DAG & Add Nodes
This part is pretty straightforward & you can use the EAC to do it. Just remember to give the DAG an IP address in every MAPI subnet where you have DAG nodes. So in our scenario above you would give the DAG 2 IP addresses; one in the 192.168.1.0 subnet & another in the 192.168.2.0 subnet.

Step 5 – Manually configure DAG Networks if needed
Reference. If you have dedicated management networks, dedicated backup networks, or iSCSI NIC’s then you would actually have to perform some manual steps after your DAG is setup. These networks should be ignored by the DAG & for cluster use. In order to do this we must first enable Manual DAG Network Configuration, which is disabled by default. We would then need to configure the iSCSI or similar network to be ignored by the cluster. Perform the following steps:

• Get-DatabaseAvailabilityGroup
• Set-DatabaseAvailabilityGroup <DAGName> -ManualDagNetworkConfiguration:$True
• Get-DatabaseAvailabilityGroupNetwork
• Set-DatabaseAvailabilityGroupNetwork <iSCSI/Backup/Mgmt NetworkName> -IgnoreNetwork:$True

Finally, let’s validate everything. Run the below command:

Get-DatabaseAvailabilityGroupNetwork | Format-List Identity,ReplicationEnabled,IgnoreNetwork

Verify that the iSCSI/Backup/Mgmt networks have IgnoreNetwork set to True (the MAPI & Replication networks should have this set to False). Also verify that the Replication Networks have ReplicationEnabled set to True. Finally, verify that the MAPI network has ReplicationEnabled set to False. This prevents the MAPI network from being used for Replication by default. It can still be used for Replication if all other possible replication paths go down.

References:

http://technet.microsoft.com/en-us/library/ff367878.aspx

http://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx

http://blogs.technet.com/b/scottschnoll/archive/2012/10/01/storage-high-availability-and-site-resilience-in-exchange-server-2013-part-2.aspx

http://blogs.technet.com/b/askcore/archive/2009/05/26/active-route-gets-removed-on-windows-2008-failover-cluster-ip-address-offline.aspx

http://technet.microsoft.com/en-us/library/dd298008(v=exchg.141).aspx

New behavior in Outlook 2013 causing certificate errors in some environments

Background:

I originally discovered this issue back in early Feb & let a couple people on the Exchange Product Team know about it via the TAP but it seems to be affecting more customers than initially thought so I thought I’d share.

In Outlook 2007 through Outlook 2010 all domain-joined Outlook clients would initially query Active Directory for AutoDiscover information & ultimately find a Service Connection Point (SCP) value that would point them to their nearest Client Access Server’s AutoDiscover virtual directory. If that failed then they would revert to using DNS like any non-domain-joined Outlook client. The order of this non-domain-joined lookup is as follows:

https://company.com/autodiscover/autodiscover.xml

https://autodiscover.company.com/autodiscover/autodiscover.xml

Local XML File

http://company.com/autodiscover/autodiscover.xml
(looking for a redirect website)

SCP AutoDiscover Record

Why it ever looked to
https://company.com/autodiscover/autodiscover.xml
I’ll never really know because honestly I’ve never come across a customer who had it deployed that way; most have
https://autodiscover.company.com/autodiscover/autodiscover.xml
but I imagine when Exchange 2007 was first being developed they weren’t exactly sure how customers would be implementing AutoDiscover.

Issue:

The above methods have served us well since Exchange 2007 timeframe but for some reason the Outlook team decided to try & implement some giddyup into Outlook & try to speed up the process. They decided to have domain-joined Outlook 2013 clients query both the SCP values in AD as well as the DNS records at the same time. If an SCP record was found it would still be used but in the event it failed then it would already have the DNS response ready to go. Great idea, however there’s one problem in the implementation.

If Outlook 2013 encounters any kind of Certificate error while doing the simultaneous DNS query then you will receive a pop-up in Outlook about the cert.

I actually stumbled upon this while in the middle of the scenario below:

That’s right, I actually get a certificate pop-up for my lab’s domain name (ash15.com) & not autodiscover.ash15.com like one would expect if I were to have a certificate issue on Exchange.

When Outlook 2013 does it’s simultaneous DNS AutoDiscover query the first URL it tries is
https://company.com/autodiscover/autodiscover.xml
, which in my lab environment resolved to my Domain Controller, which was also serving DNS, as well as a Certificate Authority. Ash15.com resolved to this server because it’s my internal Active Directory domain name & the name server entry resolves to my DC (just ping internaldomainname.local in your AD lab environment & you’ll see the same thing).

Now because I have web enrollment enabled & am listening on 443 in IIS the server responded. Also, because I did not have a cert installed on the server with ash15.com in the Subject or Subject Alternative Name then it gave the certificate error we see above.

Resolution:

The error is easy enough to get through & it only occurred on initial profile creation but this can definitely prove painful for some customers. Obviously my lab environment is a corner case but there have been several other customers report this issue with Outlook 2013 as well.

Here’s an example scenario.

Imagine you have a public website for andrewswidgets.com hosted by a third-party hosting site & you did not pay for HTTPS/443 services. However if you were to query the website using https then it could respond & obviously not return a certificate with andrewswidgets.com on it (because you haven’t paid for it you cheapskate…). Now imagine you begin deploying users using Outlook 2013 in your internal environment. In the past, they would have found the SCP record that would have pointed them to your internal Exchange 07/10/13 server for AutoDiscover & would have been happy as a clam (one Exchange Product Manager’s favorite way to describe Exchange bliss). However, now they may get a certificate pop-up for andrewswidgets.com when creating a new profile.

There are a couple ways around this. Make sure andrewswidgets.com doesn’t listen on 443, or possibly get a proper cert on your website that is listening on 443. Simply put, just make sure whatever andrewswidgets.com resolves to is something that’s not going to throw a certificate error.

I’ve heard nothing concrete or public but the Outlook team is aware of the issue & listening to customer feedback. I suggest contacting Microsoft Support if your organization is running into this issue.

 

Also, this KB offers methods to control which AutoDiscover methods are used by your Outlook clients

 

Exchange 2013 – Exchange Administration Center “Internet Explorer has stopped working” with IE 10

When you’re using RTM Server 2012 or RTM Windows 8 to manage Exchange 2013 via the Exchange Administration Center you’ll likely get a pup-up saying “Internet Explorer has stopped working”. Regardless of what option you choose IE will restart & you’ll be stuck in an endless loop of crashes, cursing, & possibly keyboard smashing.

It will typically show its ugly face when managing recipients but you may notice sporadic behavior elsewhere too.

To resolve this you’ll need to install this Microsoft Update for IE10 on Server 2012/Win8. After an install & a reboot you should be fine.

This update was actually released in December but I’m mentioning it now because I find myself building quite a few 2013 labs for self-study as well as some classes I’ll likely be teaching over the coming months. In a production environment with access to a Windows Update source this would probably go unnoticed since Windows would get updated automatically.

However, in a lab environment (with no internet access) where you’re using RTM bits for Server 2012 & Windows 8 it can become quite annoying. So I suggest either making this part of your prerequisite install list before installing 2013 or building your own OS images with it included if you plan on building lab/test environments until there are 2012/8 bits available with this fix already included.

Of course you could always just install another browser but that’s just as much of a pain in a lab as installing this KB.

Referenced KB

http://www.microsoft.com/en-us/download/details.aspx?id=35870

Microsoft Security Bulletin MS12-077 – Critical

http://technet.microsoft.com/en-us/security/bulletin/ms12-077

Exchange 2013 Gotchas

http://theessentialexchange.com/blogs/michael/archive/2013/01/06/exchange-server-2013-gotchas.aspx

External Connections to Internal Web Servers Over 443 Fail

Probably one of the strangest issues I’ve seen, at least it seemed that way at the time.

Scenario:

2 internal web servers experiencing the symptom, one running 2010 OWA and the other a custom web application on 443. All internal users can hit each page just fine. External users cannot hit the pages and they just receive a timeout. However, if the admin logs into either of the two servers locally or via RDP and then you try again externally, it works and they can hit the web pages. This behavior only happened on 443. Customer was just using a Cisco ASA for their firewall with no web publishing.

Resolution:

Customer was a school district and I was reminded of a former life where I worked for a school district where web filtering was common. We found out that external users could only hit the pages when an Admin was logged into either of the servers; not a regular user. Combined with the below Cisco thread I found when trying to potentially pin this on the ASA it seemed a web filter or Intrusion Detection System was killing our connections.

https://supportforums.cisco.com/thread/2043090

According to the thread a Filter/IDS on the inside could potentially be issuing resets for web traffic that it did not like. In our case it was the customers “iBoss” content filter that started blocking access after a firmware update. It worked when an Administrator was logged into the web servers because it could filter based on the currently logged on AD account and there were exclusions for the Admins.

Getting Lync PowerShell to use RBAC

You may find that some things will work in the Lync GUI that will not work in PowerShell (Access Denied), the reason for this is that RBAC only applies to remote PowerShell and local PowerShell uses the AD permissions and not RBAC.

To resolve this you can login to PowerShell using the following script: (Copy the contents to a file and name it Connect-Lync.ps1)

$usercredential = get-credential
$pso = new-pssessionoption -skipcacheck -SkipCNCheck -SkipRevocationCheck
$session= New-PSSession -ConnectionUri
https://localhost/ocspowershell
-credential $usercredential -sessionoption $pso
import-pssession $session

Note: 1. This script ignores the certificate (so it will work if your using a self signed cert)
          2. You may need to modify the execution policy to run this unsigned script in PowerShell “set-executionpolicy remote”

References:

http://technet.microsoft.com/en-us/library/gg399050.aspx

“Note that RBAC applies only to remote management. If you are logged on to a computer running Lync Server 2010 and you open Lync Server Management Shell, RBAC roles will not be enforced. Instead, security is enforced primarily through the security groups RTCUniversalServerAdmins; RTCUniversalUserAdmins; and RTCUniversalReadOnlyAdmins.”

The Problem with Hardware VSS Providers and Cluster Technologies like CSV and DAG

In solutions like DAG and CSV you can have issues with VSS backups completing if you are attached to a SAN and using a hardware provider.
The reason for this is because the LUN needs to pause the processes accessing the LUN but if another server is the one in control  of data on that LUN its unable to do that on a single host.
Here are some details as well as ways to resolve this issue.

Scenarios:

1. CSV Issue

• Multiple Servers with a shared CSV Volume and VMS distributed across nodes may fail if you are using hardware VSS providers because it wants to snapshot the entire LUN but the node you are running the snap shot from doesn’t have access to all the VMS in order to pause them before committing the snapshot.

1. You can resolve this in one of 2 ways.
   1. Move all the VMs to a single node or host until the backup is completed.
   2. Disable or remove your hardware based VSS provider.

 

 

 

2. DAG Issue

This issue may come up not because you are sharing LUNS and have active data  on separate nodes (as above) but because you may use a separate provider for Active and Passive backups. When you try to backup a LUN that has both active and passive databases a hardware provider may try to use two different writers to snapshot the LUN. You can verify this by moving all active databases to one node to backup.

1. You can resolve this in one of 3 ways.
   1. Do not put multiple databases on a single LUN.
   2. Move all Databases to one node before running backup
2. 3. Disable you hardware based VSS provider

 

NOTE: Disabling your hardware provider will likely cause your backups to take much longer

References

• Disable Equal Logic VSS Writer – Run C:\Program Files\EqualLogic\bin>eqlvss /unregserver”
• Disable Hardware VSS in DPM – Add the following key to the registry [Software\Microsoft\Microsoft Data Protection Manager\Agent\UseSystemSoftwareProvider]
• How VSS Works
• If you know how to disable other providers please let me know and I will add it to this document!

Configure Split DNS for a specific Host

 

Say I have domain.com and its hosted externally.
I add an exchange server and, I add an external record pointing to my server called mail.domain.com and it points to my external IP.
I ALSO want to be able to access my server using the internal IP instead of going through my firewall and back in. (This is called split DNS)

Split DNS = I have 2 DNS zones, one external and one internal for the same domain.
The issue is that you have to manage both zones individually (even if you only need one specific host record)

And alternative method is to create a zone JUST for that one host name.

Here are the directions to create a domain and same as parent A record

1. Open DNS on your DC, right click Forward Lookup Zone, and select  New Zone
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
10. 
11. 

Now you have split DNS for the single host name only.

Exchange 2010 EMC cannot access AD configuration data after you demote a DC

Environment:

Exchange 2010/Domain Controller combo server running on Windows 2008 R2.

Problem:

Demote Domain Controller role, causes Exchange Management Console fails to retrieve any Exchange information with error message “Active directory response: The LDAP server is unavailable.”  It’s still looking for the demoted DC although it’s been cleaned out of AD/DNS. All Exchange services start fine, and Exchange Shell works fine.

Reason:

     The obsolete information is cached in an Exchange Management Console file in the Windows profile for the user. EMC is trying to connect to orginal DC that is stored in the file.

Solution:

   Go to the following folder and delete the Exchange Management Console file.

   C:\users\<specific user>\AppData\Roaming\Microsoft\MMC\Exchange Management Console

   Close EMC and reopen it.

Reference:
http://support.microsoft.com/kb/2019500

/Preparedomain error when security customizations have been done to Active Directory

Error:

Configuring Microsoft Exchange Server

    Organization Preparation                                  FAILED
     The following error was generated when "$error.Clear();
          if ($RolePrepareAllDomains)
          {
              initialize-DomainPermissions -AllDomains:$true -CreateTenantRoot:(
$RoleIsDatacenter -or $RoleIsPartnerHosted);
          }
          elseif ($RoleDomain -ne $null)
          {
              initialize-DomainPermissions -Domain $RoleDomain -CreateTenantRoot
$RoleIsDatacenter -or $RoleIsPartnerHosted);
          }
          else
          {
              initialize-DomainPermissions -CreateTenantRoot:($RoleIsDatacenter
-or $RoleIsPartnerHosted);
          }
        " was run: "PrepareDomain for domain Domain was unable to add the group CN=Exchange Install Domain Servers,CN=Microsoft Exchange System Objects,DC=domain,DC=local to the group CN=Exchange Servers,OU=Microsoft Exchange Security Groups,DC=domain,DC=local on domain controller server.domain.local, because the current user does not have permissions to modify Exchange Servers. Please ensure that the current user can modify the membership of Exchange Servers and run PrepareDomain again.".

Problem:
The user doesn’t have permission to modify the AD groups it needs to modify.
“Exchange Server” group that was created by /preparedomain is member of “Windows Authorization Access Group” group. 
If the permission on that group are changed, /preparedomain may not be able to modify the membership of it. 
Of course, exchange setup gives you some bogus error, which does not make any sense.

Solution:

1. Verify that you are running the /preparedomain as a domain admin
2. Once we reset it’s permission by checking “inherted” option on the “Windows Authorization Access Group”,  we can manually add Exchange Server group as a member of “Windows Authorization Access Group” Group, and re run /preparedomain and it should run without error.