Monday, February 23, 2015

VSS Snapshot error. The maximum number of snapshots for this volume has been reached.

Today on a customers SBS 2008 server still running Backup Exec 2010, backups started failing with the following error.

 - AOFO: Initialization failure on: "\\JCC-SBS\Microsoft Information Store\First Storage Group". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11234 - VSS Snapshot error. The maximum number of snapshots for this volume has been reached.  Microsoft Volume Shadow Copy Service (VSS)  will not allow any more snapshots.

The Advanced Open File Option was set to Automatically select open file technology - I changed this to use "System - Use Microsoft Software Shadow Copy Provider".


After making this change, I tested another backup and it completed successfully.


Interestingly, if you read the error it was already using the Microsoft Volume Shadow Provider yet it failed.  Hardcoding it in the settings of the backup job did the trick.

Phantom Calls to Cisco Phone with Hosted VOIP Service - How to Lockdown

An issue was experienced where a number of VOIP phones on a customer network were experiencing phantom calls from the Internet.  A phantom call generally occurs when someone is scanning for SIP devices on a network using a port scanning program.  A port scan against SIP (usually UDP 5060 and 5061) will cause many SIP handsets to annoyingly ring and when the user answers, no one is there!

As you could gather, this is very annoying for users on the network.

The solution for this is locking down SIP to only the Public IP addresses of your SIP IP Gateway from your VOIP provider on the public interface of the router.  This customer was using a Cisco 887va router connected on a DSL connection so the public interface in this instance is dialer0.

First of all you need to create a access rule.  Here is the Access Rule I created (this matches the Faktortel VOIP providers public IP addresses here in Australia).  Faktortel also use 5062, 5063, 5064 and 5065 UDP ports for SIP communication.

ip access-list extended VOIPLockDown
remark allow SIP from certain addresses
permit udp host 202.43.66.55 any eq 5060
permit udp host 202.43.66.55 any eq 5061
permit udp host 202.43.66.55 any eq 5062
permit udp host 202.43.66.55 any eq 5063
permit udp host 202.43.66.55 any eq 5064
permit udp host 202.43.66.55 any eq 5065
permit udp host 202.43.66.1 any eq 5060
permit udp host 202.43.66.1 any eq 5061
permit udp host 202.43.66.1 any eq 5062
permit udp host 202.43.66.1 any eq 5063
permit udp host 202.43.66.1 any eq 5064
permit udp host 202.43.66.1 any eq 5065
permit udp host 202.43.66.2 any eq 5060
permit udp host 202.43.66.2 any eq 5061
permit udp host 202.43.66.2 any eq 5062
permit udp host 202.43.66.2 any eq 5063
permit udp host 202.43.66.2 any eq 5064
permit udp host 202.43.66.2 any eq 5065
permit udp host 202.43.66.3 any eq 5060
permit udp host 202.43.66.3 any eq 5061
permit udp host 202.43.66.3 any eq 5062
permit udp host 202.43.66.3 any eq 5063
permit udp host 202.43.66.3 any eq 5064
permit udp host 202.43.66.3 any eq 5065
permit udp host 202.43.66.4 any eq 5060
permit udp host 202.43.66.4 any eq 5061
permit udp host 202.43.66.4 any eq 5062
permit udp host 202.43.66.4 any eq 5063
permit udp host 202.43.66.4 any eq 5064
permit udp host 202.43.66.4 any eq 5065
permit udp host 202.43.66.5 any eq 5060
permit udp host 202.43.66.5 any eq 5061
permit udp host 202.43.66.5 any eq 5062
permit udp host 202.43.66.5 any eq 5063
permit udp host 202.43.66.5 any eq 5064
permit udp host 202.43.66.5 any eq 5065
remark
remark block SIP from all other addresses
deny udp any any eq 5060
deny udp any any eq 5061
deny udp any any eq 5062
deny udp any any eq 5063
deny udp any any eq 5064
deny udp any any eq 5065
remark
remark Allow all other IP traffic
permit ip any any


This rule permits all SIP traffic to all of their public IP addresses then blocks all other traffic entering on these ports and finally permits any/any.

Next you need to assign the access rule to the public interface on the router you wish to filter.  Because this is a Cisco 887va router on DSL, the public interface is Dialer0.  Simply apply the access rule to Dialer0 as follows:

interface dialer0
ip access-group VOIPLockDown in


After applying this config change, all phantom calls to phones on the network stopped.

Important: Some SIP providers use both TCP and UDP for SIP, so it can be useful to block both UDP and TCP on these port numbers.  For this customer, the SIP handsets only supported UDP so there was no point blocking the TCP ports.  If your not sure, block both by slightly modifying the config above.

Tuesday, February 3, 2015

Direct Access Clients Cannot Establish Network Connectivity after NLS Becomes Unavailable

In Microsoft Direct Access Deployments, you need to configure a server (or server cluster) on your network to act as a Network Location Server (NLS).  The NLS is simply a Web Server with a HTTPS server certificate... see the following TechNet article on how to configure an NLS:

https://technet.microsoft.com/en-us/library/ee649252(v=ws.10).aspx

If the DirectAccess client can establish a connection to the NLS, it will believe that it is on the internal network and the DirectAccess tunnels will not be established. If a connection to the NLS cannot be established, the DirectAccess client believes it is outside the corporate network and will attempt to establish DirectAccess connectivity. It is for this reason that the NLS should not be resolvable in public DNS and should not be reachable externally. If it is, DirectAccess clients will always think they are inside the corporate network, and DirectAccess will never be established. Also, it is extremely important that the NLS be highly available. If the NLS server is offline or unreachable for any reason, DirectAccess clients that are on the internal network will suddenly think they are outside the corporate network and attempt to establish DirectAccess connectivity. This will fail, leaving DirectAccess clients unable to connect to any internal resources until the NLS is once again available.  For this reason it is strongly recommended that you implement at least two NLS servers in a cluster, using either Network Load Balancing (NLB) or an external hardware load balancer.

However for smaller deployments of Direct Access, a NLS cluster for redundancy is not always practical and sometimes only a single NLS server is deployed.  This is often the case with all in one Direct Access Deployments running Windows Server 2012 R2 when a single Direct Access server is deployed to act as both the NLS and Direct Access endpoint.  If the all in one Direct Access server was to fail, all clients on the Internal network will no longer think they are on the internal network as they cannot contact the NLS and as a result attempt to establish a Direct Access connection which will also fail leaving them in a state of no connectivity to any domain names configured with a Name Resolution Policy Table (NRPT) in the Direct Access group policy.  Most Direct Access deployments have the Active Directory domain namespace configured as an NRPT policy and sometimes additional domain names.  To understand what NRPT is, please see the following blog post:

http://clintboessen.blogspot.com.au/2014/01/what-is-name-resolution-policy-table.html

What Happens if Your In this Situation?

What happens if your in this situation when you have a single Direct Access server which has failed and as a result all Direct Access configured clients on the internal network now have no network connectivity?

The NRPT Policies which are pushed out via Group Policy are located under the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\DnsClient\DnsPolicyConfig

You can manually connect to the direct access computers and remove these registry keys by IP address using the "Connect Network Registry" option in regedit.exe.  Once you are connected to the remote computer, you can delete the two NRPT policies named DA-{GUID].


After you have deleted the two policies remotely, simply reboot the workstation and it will regain connectivity.

I know this is a very manual process and can be very time consuming if there are hundreds of computers experiencing the issue - I would like to hear from Microsoft for a better solution in this scenario.  I sent of an email regarding this and hopefully will hear back soon!

It is also important to note, for my customer who's Direct Access server failed, rebuilding a new server with the same IP address, hostname and configuration did not resolve this issue.  I still needed to delete the NRPT policies from the registry from the effected computers and reboot.  After this, the clients had restored connectivity to the internal network and were able to refresh group policy again in which they downloaded the new Direct Access Client Settings containing the new NRPT policies.