Hello,
We have a two-leg (DMZ+LAN) Direct Access infrastructure setup on Windows Server 2012 with Win 7 computers clients. Out of sudden, after the last server restart it stopped working. IP Sec Main mode negotiations are failing on both ends(client & server) with event id 4653. Can somebody help me troubleshoot this IPSec errors?
Looking with tcpview, I can see on both sides that the connection to https port on DA server is established.
On server side, I see only green checkmarks in Remote Access console. There are times when Network Security module is reporting that is under a DOS attack (probably caused by the high number of connections ~1000 that are failing IPSec)
A wireshark trace is showing ipv6 traffic only in one direction, from fd00:0:0:1000::1 toward the remote client. I cannot see anything where the source is the ipv6 address of client.
- On the client side, I also get 4563 event ids:
The IPHTTPS interface is reporting as active, but it cannot reach the DA, DNS or any other infrastructure server.
DirectAccess Client Troubleshooter Tool is reporting:
[28/11/2018 10:37:30]: In worker thread, going to start the tests.
[28/11/2018 10:37:30]: Running Network Interfaces tests.
[28/11/2018 10:37:30]: Wireless Network Connection (Intel(R) Centrino(R) Advanced-N 6205): 10.3.77.53/255.255.252.0;
[28/11/2018 10:37:30]: Default gateway found for Wireless Network Connection.
[28/11/2018 10:37:30]: iphttpsinterface (iphttpsinterface): fd00::1000:4005:d6ac:8164:5a85;: fd00::1000:f45f:b394:7649:f2f9;: fe80::4005:d6ac:8164:5a85%18;
[28/11/2018 10:37:30]: No default gateway found for iphttpsinterface.
[28/11/2018 10:37:30]: Wireless Network Connection has configured the default gateway 10.3.79.254.
[28/11/2018 10:37:42]: Warning - default gateway 10.3.79.254 for Wireless Network Connection does not reply on ICMP Echo requests, the request or response is maybe filtered?
[28/11/2018 10:37:42]: Received a response from the public DNS server (8.8.8.8), RTT is 41 msec.
[28/11/2018 10:37:42]: The public DNS Server (2001:4860:4860::8888) does not reply on ICMP Echo requests, the request or response is maybe filtered?
[28/11/2018 10:37:42]: Running Inside/Outside location tests.
[28/11/2018 10:37:42]: NLS is https://nls.<COMPANY>.local/.
[28/11/2018 10:37:42]: NLS is not reachable via HTTPS, the client computer is not connected to the corporate network (external) or the NLS is offline.
[28/11/2018 10:37:42]: NRPT contains 3 rules.
[28/11/2018 10:37:42]: Found (unique) DNS server: fd00::a03:ea
[28/11/2018 10:37:42]: Send an ICMP message to check if the server is reachable.
[28/11/2018 10:37:54]: DNS Server fd00::a03:ea does not reply on ICMP Echo requests.
[28/11/2018 10:37:54]: Running IP connectivity tests.
[28/11/2018 10:37:54]: The 6to4 interface service state is default.
[28/11/2018 10:37:54]: Teredo inferface status is offline.
[28/11/2018 10:37:54]: The configured Teredo server is the public Microsoft Teredo server teredo.ipv6.microsoft.com..
[28/11/2018 10:37:54]: The IPHTTPS interface is operational.
[28/11/2018 10:37:54]: The IPHTTPS interface status is IPHTTPS interface active.
[28/11/2018 10:37:54]: IPHTTPS is used as IPv6 transition technology.
[28/11/2018 10:37:54]: The configured IPHTTPS URL is https://da.<COMPANY>.com:443.
[28/11/2018 10:37:54]: IPHTTPS has a single site configuration.
[28/11/2018 10:37:54]: IPHTTPS URL endpoint is: https://da.<COMPANY>.com:443.
[28/11/2018 10:37:55]: Successfully connected to endpoint https://da.<COMPANY>.com:443.
[28/11/2018 10:37:55]: No response received from <COMPANY>.local.
[28/11/2018 10:37:55]: Running Windows Firewall tests.
[28/11/2018 10:37:55]: The current profile of the Windows Firewall is Public.
[28/11/2018 10:37:55]: The Windows Firewall is enabled in the current profile Public.
[28/11/2018 10:37:55]: The outbound Windows Firewall rule Core Networking - Teredo (UDP-Out) is enabled.
[28/11/2018 10:37:55]: The outbound Windows Firewall rule Core Networking - IPHTTPS (TCP-Out) is enabled.
[28/11/2018 10:37:55]: Running certificate tests.
[28/11/2018 10:37:55]: Found 1 machine certificates on this client computer.
[28/11/2018 10:37:55]: Checking certificate [no subject] with the serial number [15CF7D9B0005000094D7].
[28/11/2018 10:37:55]: The certificate [15CF7D9B0005000094D7] contains the EKU Client Authentication.
[28/11/2018 10:37:57]: The trust chain for the certificate [15CF7D9B0005000094D7] was sucessfully verified.
[28/11/2018 10:37:57]: Running IPsec infrastructure tunnel tests.
[28/11/2018 10:37:57]: Failed to connect to domain sysvol share \\<COMPANY>.local\sysvol\<COMPANY>.local\Policies.
[28/11/2018 10:37:57]: Running IPsec intranet tunnel tests.
[28/11/2018 10:38:09]: Failed to connect to fd00:0:0:1000::1 with status TimedOut.
[28/11/2018 10:38:21]: Failed to connect to fd00:0:0:1000::2 with status TimedOut.
[28/11/2018 10:38:21]: Failed to connect to HTTP probe at http://directaccess-WebProbeHost.<COMPANY>.local.
[28/11/2018 10:38:21]: Running selected post-checks script.
[28/11/2018 10:38:21]: No post-checks script specified or the file does not exist.
[28/11/2018 10:38:21]: Finished running post-checks script.
[28/11/2018 10:38:21]: Finished running all tests.
Below is the output from some common troubleshooting commands:
<CMD>ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : <HOSTNAME>
Primary Dns Suffix . . . . . . . : <COMPANY>.local
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : <COMPANY>.local
Ethernet adapter Bluetooth Network Connection:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Bluetooth Device (Personal Area Network)
Physical Address. . . . . . . . . : C0-F8-DA-E3-1B-90
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Wireless LAN adapter Wireless Network Connection:
Connection-specific DNS Suffix . : <COMPANY>.local
Description . . . . . . . . . . . : Intel(R) Centrino(R) Advanced-N 6205
Physical Address. . . . . . . . . : A0-88-B4-55-F8-F0
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 10.3.77.53(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.252.0
Lease Obtained. . . . . . . . . . : 28 November 2018 08:59:53
Lease Expires . . . . . . . . . . : 02 December 2018 10:00:00
Default Gateway . . . . . . . . . : 10.3.79.254
DHCP Server . . . . . . . . . . . : 10.3.80.1
DNS Servers . . . . . . . . . . . : 8.8.8.8
NetBIOS over Tcpip. . . . . . . . : Enabled
Tunnel adapter isatap.{4A3D349D-D1ED-4F0E-967F-D4612C286083}:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft ISATAP Adapter
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Tunnel adapter isatap.<COMPANY>.local:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . : <COMPANY>.local
Description . . . . . . . . . . . : Microsoft ISATAP Adapter #3
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Tunnel adapter iphttpsinterface:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : iphttpsinterface
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : fd00::1000:4005:d6ac:8164:5a85(Preferred)
Temporary IPv6 Address. . . . . . : fd00::1000:f45f:b394:7649:f2f9(Preferred)
Link-local IPv6 Address . . . . . : fe80::4005:d6ac:8164:5a85%18(Preferred)
Default Gateway . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled
<CMD>Netsh dnsclient show state
Name Resolution Policy Table Options
--------------------------------------------------------------------
Query Failure Behavior : Always fall back to LLMNR
and NetBIOS
if the name does not
exist in DNS or
if the DNS servers
are unreachable
when on a private
network
Query Resolution Behavior : Resolve only IPv6 addresses for names
Network Location Behavior : Let Network ID determine when Direct
Access settings are
to be used
Machine Location : Outside corporate network
Direct Access Settings : Configured and Enabled
DNSSEC Settings : Not Configured
<CMD>Netsh interface httpstunnel show interface
Interface IPHTTPSInterface (Group Policy) Parameters
------------------------------------------------------------
Role : client
URL :https://da.<COMPANY>.com:443/IPHTTPS
Last Error Code : 0x0
Interface Status : IPHTTPS interface active
<CMD>Netsh namespace show effectivepolicy
DNS Effective Name Resolution Policy Table Settings
Settings for nls.<COMPANY>.local
----------------------------------------------------------------------
Certification authority :
DNSSEC (Validation) : disabled
IPsec settings : disabled
DirectAccess (DNS Servers) :
DirectAccess (Proxy Settings) : Use default browser settings
Settings for .<COMPANY>.local
----------------------------------------------------------------------
Certification authority :
DNSSEC (Validation) : disabled
IPsec settings : disabled
DirectAccess (DNS Servers) : fd00::a03:ea
DirectAccess (Proxy Settings) : Bypass proxy
<CMD>Netsh advfirewall monitor show mmsa
No SAs match the specified criteria.
<CMD>Netsh advfirewall show currentprofile
Public Profile Settings:
----------------------------------------------------------------------
State ON
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Enable
RemoteManagement Disable
UnicastResponseToMulticast Enable
Logging:
LogAllowedConnections Enable
LogDroppedConnections Enable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 24096
Ok.
<CMD>Certutil -store my
my
================ Certificate 0 ================
Serial Number: 61d68c3200050000946c
Issuer: CN=<COMPANY-CAName>, DC=<COMPANY>, DC=Local
NotBefore: 15/11/2018 13:35
NotAfter: 02/07/2019 11:43
Subject: EMPTY (DNS Name=<HOSTNAME>.<COMPANY>.local)
Non-root Certificate
Template: 1.3.6.1.4.1.311.21.8.6693252.4963786.7359385.10098729.16443910.70.7655005.1833759
Cert Hash(sha1): fb 5d d5 b2 31 57 83 bb 9b 68 b8 91 b8 f2 b2 a4 8b a2 51 ac
Key Container = f588ece0f8e5701064bc0b40d7c606f2_704c463e-1552-49a5-8244-f045c492456d
Simple container name: le-SCCMClientCertificate-f86fdc58-1726-4779-9be3-aa3023c0fa21
Provider = Microsoft RSA SChannel Cryptographic Provider
Private key is NOT exportable
Encryption test passed
On the server side, I've performed a restore back in time to a point when I know for sure that DA was working. Along with this, I've also restored the DA GPOs. This has not helped, so it makes me think that the issue is not on the DA server itself.
I don't believe that it can be on client side, as I consider that if it was this case, at least I could have seen at least one connected client. Or maybe the DOS protection of DA server is preventing all client connections.
Does anyone have any idea what might be wrong?