Probably one of the strangest issues I’ve seen, at least it seemed that way at the time.
2 internal web servers experiencing the symptom, one running 2010 OWA and the other a custom web application on 443. All internal users can hit each page just fine. External users cannot hit the pages and they just receive a timeout. However, if the admin logs into either of the two servers locally or via RDP and then you try again externally, it works and they can hit the web pages. This behavior only happened on 443. Customer was just using a Cisco ASA for their firewall with no web publishing.
Customer was a school district and I was reminded of a former life where I worked for a school district where web filtering was common. We found out that external users could only hit the pages when an Admin was logged into either of the servers; not a regular user. Combined with the below Cisco thread I found when trying to potentially pin this on the ASA it seemed a web filter or Intrusion Detection System was killing our connections.
According to the thread a Filter/IDS on the inside could potentially be issuing resets for web traffic that it did not like. In our case it was the customers “iBoss” content filter that started blocking access after a firmware update. It worked when an Administrator was logged into the web servers because it could filter based on the currently logged on AD account and there were exclusions for the Admins.
You may find that some things will work in the Lync GUI that will not work in PowerShell (Access Denied), the reason for this is that RBAC only applies to remote PowerShell and local PowerShell uses the AD permissions and not RBAC.
To resolve this you can login to PowerShell using the following script: (Copy the contents to a file and name it Connect-Lync.ps1)
$usercredential = get-credential
$pso = new-pssessionoption -skipcacheck -SkipCNCheck -SkipRevocationCheck
$session= New-PSSession -ConnectionUri https://localhost/ocspowershell -credential $usercredential -sessionoption $pso
Note: 1. This script ignores the certificate (so it will work if your using a self signed cert)
2. You may need to modify the execution policy to run this unsigned script in PowerShell “set-executionpolicy remote”
“Note that RBAC applies only to remote management. If you are logged on to a computer running Lync Server 2010 and you open Lync Server Management Shell, RBAC roles will not be enforced. Instead, security is enforced primarily through the security groups RTCUniversalServerAdmins; RTCUniversalUserAdmins; and RTCUniversalReadOnlyAdmins.”
In solutions like DAG and CSV you can have issues with VSS backups completing if you are attached to a SAN and using a hardware provider.
The reason for this is because the LUN needs to pause the processes accessing the LUN but if another server is the one in control of data on that LUN its unable to do that on a single host.
Here are some details as well as ways to resolve this issue.
1. CSV Issue
- Multiple Servers with a shared CSV Volume and VMS distributed across nodes may fail if you are using hardware VSS providers because it wants to snapshot the entire LUN but the node you are running the snap shot from doesn’t have access to all the VMS in order to pause them before committing the snapshot.
- You can resolve this in one of 2 ways.
1. Move all the VMs to a single node or host until the backup is completed.
2. Disable or remove your hardware based VSS provider.
2. DAG Issue
This issue may come up not because you are sharing LUNS and have active data on separate nodes (as above) but because you may use a separate provider for Active and Passive backups. When you try to backup a LUN that has both active and passive databases a hardware provider may try to use two different writers to snapshot the LUN. You can verify this by moving all active databases to one node to backup.
- You can resolve this in one of 3 ways.
1. Do not put multiple databases on a single LUN.
2. Move all Databases to one node before running backup
- 3. Disable you hardware based VSS provider
NOTE: Disabling your hardware provider will likely cause your backups to take much longer
- Disable Equal Logic VSS Writer – Run C:\Program Files\EqualLogic\bin>eqlvss /unregserver”
- Disable Hardware VSS in DPM – Add the following key to the registry [Software\Microsoft\Microsoft Data Protection Manager\Agent\UseSystemSoftwareProvider]
- How VSS Works
- If you know how to disable other providers please let me know and I will add it to this document!
Exchange 2010/Domain Controller combo server running on Windows 2008 R2.
Demote Domain Controller role, causes Exchange Management Console fails to retrieve any Exchange information with error message “Active directory response: The LDAP server is unavailable.” It’s still looking for the demoted DC although it’s been cleaned out of AD/DNS. All Exchange services start fine, and Exchange Shell works fine.
The obsolete information is cached in an Exchange Management Console file in the Windows profile for the user. EMC is trying to connect to orginal DC that is stored in the file.
Go to the following folder and delete the Exchange Management Console file.
C:\users\<specific user>\AppData\Roaming\Microsoft\MMC\Exchange Management Console
Close EMC and reopen it.
Configuring Microsoft Exchange Server
Organization Preparation FAILED
The following error was generated when "$error.Clear();
initialize-DomainPermissions -AllDomains:$true -CreateTenantRoot:(
$RoleIsDatacenter -or $RoleIsPartnerHosted);
elseif ($RoleDomain -ne $null)
initialize-DomainPermissions -Domain $RoleDomain -CreateTenantRoot
$RoleIsDatacenter -or $RoleIsPartnerHosted);
" was run: "PrepareDomain for domain Domain was unable to add the group CN=Exchange Install Domain Servers,CN=Microsoft Exchange System Objects,DC=domain,DC=local to the group CN=Exchange Servers,OU=Microsoft Exchange Security Groups,DC=domain,DC=local on domain controller server.domain.local, because the current user does not have permissions to modify Exchange Servers. Please ensure that the current user can modify the membership of Exchange Servers and run PrepareDomain again.".
The user doesn’t have permission to modify the AD groups it needs to modify.
“Exchange Server” group that was created by /preparedomain is member of “Windows Authorization Access Group” group.
If the permission on that group are changed, /preparedomain may not be able to modify the membership of it.
Of course, exchange setup gives you some bogus error, which does not make any sense.
- Verify that you are running the /preparedomain as a domain admin
- Once we reset it’s permission by checking “inherted” option on the “Windows Authorization Access Group”, we can manually add Exchange Server group as a member of “Windows Authorization Access Group” Group, and re run /preparedomain and it should run without error.
OK this is a twilight zone issue.
We had a situation where a 2008 R2 server has very slow download\transfer speeds across a WAN\ Internet on a new server using 2008 R2, if we change the OS its all good and if we do another server its all good.
- We investigated the network looking for issues with switch and routers
- Disabled all advanced network protocols
- netsh int tcp show global to view
- netsh int tcp set global <parameter>=disabled
- disabled all from NIC properties –> configure
- various other bang your head on the wall
- Lots o research
- Found some info indicating that on the Win 7\R2 Kernel if you have more memory than the OS supports it can cause a related issue.
Remove all memory over what the OS supports: i.e 2008 R2 standard = 32gb/
It sounds totally weird but its true.