Troubleshooting SDN on Windows Server: A Practical Guidance

Table of Contents

This document provides practical guidance for troubleshooting Software Defined Networking (SDN) technologies within Windows Server environments, specifically versions 2019, 2016, and Azure Stack HCI. The aim is to empower users to independently diagnose and resolve common SDN issues encountered in these environments, minimizing downtime and ensuring smooth network operations. The solutions outlined are designed to address the majority of user-reported problems, offering a structured approach to SDN troubleshooting.

Troubleshooting Checklist¶

When encountering issues within your SDN infrastructure, a systematic checklist is crucial for efficient diagnosis. This checklist focuses on key areas of SDN configuration and connectivity, guiding you through a step-by-step process to pinpoint the root cause of the problem. By methodically examining each component, you can isolate the source of the issue and apply targeted solutions, significantly reducing troubleshooting time.

Check IP Configuration and Virtual Subnets Referencing ACL¶

Ensuring correct IP configuration and virtual subnet settings is fundamental to SDN functionality. Access Control Lists (ACLs) heavily rely on accurate IP addressing and subnet definitions to enforce network policies effectively. Inconsistencies or errors in these configurations can lead to connectivity problems and policy enforcement failures. Therefore, verifying these settings is a critical initial step in SDN troubleshooting.

1. Verify Provider Address and Connectivity: Begin by executing the Get-ProviderAddress command on both Hyper-V hosts involved with the affected tenant virtual machines (VMs). This command retrieves the Provider Addresses (PAs) assigned to these hosts within the SDN overlay network. Subsequently, use Test-LogicalNetworkConnection or ping -c <compartment> from the Hyper-V host to validate network connectivity on the HNV Provider logical network. Successful connectivity at this level is essential for the VMs to communicate within the SDN environment.

2. Validate MTU Settings: Incorrect Maximum Transmission Unit (MTU) settings can severely impact network performance and even disrupt connectivity. MTU defines the largest packet size that can be transmitted over a network. In SDN environments, especially those using encapsulation technologies like VXLAN or NVGRE, overhead is added to packets, requiring a larger MTU to accommodate. Execute Test-EncapOverheadValue on all affected Hyper-V hosts to determine the encapsulation overhead. Critically, ensure that all Layer-2 switches between the Hyper-V hosts are configured with an MTU of at least 1,674 bytes. This accounts for a maximum encapsulation overhead of 160 bytes, preventing packet fragmentation and ensuring efficient data transmission.

3. Inspect Network Policy Reception: If Provider Addresses (PAs) are absent, or if Customer Address (CA) connectivity is disrupted, the issue might stem from incomplete or incorrect network policy distribution. Utilize the Get-PACAMapping command to examine whether encapsulation rules and CA-PA mappings, which are essential for creating overlay virtual networks, have been correctly established. These mappings are the cornerstone of SDN’s ability to isolate and manage tenant networks, so their proper configuration is paramount.

4. Confirm Network Controller Host Agent Connection: The Network Controller Host Agent acts as a crucial communication bridge between the Hyper-V host and the central Network Controller. Verify that this agent is actively connected to the Network Controller. Employ the command netstat -anp tcp | findstr 6640 to check for established TCP connections on port 6640, the default port used by the Network Controller Host Agent. A healthy connection is indicated by an established state, ensuring that the Hyper-V host can receive and implement network policies from the controller.

5. Registry Host ID Verification: The Host ID, stored in the HKLM registry key, must accurately match the Instance ID of the server resources hosting the tenant VMs. This ID serves as a unique identifier for the Hyper-V host within the SDN infrastructure. Mismatches can lead to policy misapplication or communication failures. Carefully compare these IDs to ensure consistency and proper host identification.

6. Port Profile ID Consistency: Similarly, the Port Profile ID should align with the Instance ID of the VM network interfaces of the tenant VMs. The Port Profile defines the network policies and configurations applied to a VM’s network interface. Discrepancies between these IDs can result in VMs not receiving the correct network policies, leading to connectivity and security issues. Verify the Port Profile IDs to guarantee that each VM network interface is correctly associated with its intended policy profile.

Check Network Connectivity Between the Network Controller and Hyper-V Host¶

Robust network connectivity between the Network Controller and Hyper-V hosts is indispensable for SDN to function correctly. The Network Controller centrally manages and distributes network policies, and Hyper-V hosts rely on this connection to receive and enforce these policies. Interrupted or unstable connectivity will disrupt SDN operations and lead to policy enforcement failures.

Use the netstat command to scrutinize the connections between the Network Controller (NC) Host Agent and the Network Controller nodes. A healthy connection state should exhibit the following characteristics:

• Listening Port on Hyper-V Host: Port TCP:6640 should be in a LISTENING state on the Hyper-V Host. This signifies that the NC Host Agent service is actively running and ready to accept connections from the Network Controller.
• Established Connections to NC Nodes: There should be two established connections originating from the Hyper-V host IP on port 6640 to the Network Controller node IPs on ephemeral ports (ports numbered higher than 32000). These represent the primary communication channels initiated by the Host Agent to the Network Controller for policy updates and status reporting.
• Established Connection to NC REST IP: One established connection should exist from the Hyper-V host IP on an ephemeral port to the Network Controller REST IP on port 6640. This connection is used for REST API communication, allowing the Host Agent to interact with the Network Controller’s management interface.

Check Host Agent Services¶

The Network Controller relies on two crucial host agent services on each Hyper-V host: the SLB Host Agent and the NC Host Agent. These agents act as intermediaries, facilitating communication and policy enforcement on the host level. If either of these services is not running, SDN functionality on that host will be severely impaired.

Verify the status of these services and restart them if they are not running. Use the following PowerShell commands:

Get-Service SlbHostAgent
Get-Service NcHostAgent

# Start services if not running
Start-Service NcHostAgent
Start-Service SlbHostAgent

Check Health of Network Controller¶

The Network Controller’s health is paramount to the overall SDN infrastructure stability. If the Network Controller is unhealthy or unresponsive, it cannot effectively manage network policies, leading to widespread issues. Therefore, monitoring and verifying the Network Controller’s health is a critical troubleshooting step.

If you observe fewer than three ESTABLISHED connections as described previously, or if the Network Controller appears unresponsive, investigate the health of its nodes and service modules. Utilize the Debug-ServiceFabricNodeStatus cmdlet to examine the status of individual service module replicas:

# Prints the status of a specific service module replica
Debug-ServiceFabricNodeStatus [-ServiceTypeName] <Service Module>

The essential Network Controller service modules to monitor include:

• ControllerService
• ApiService
• SlbManagerService
• ServiceInsertion
• FirewallService
• VSwitchService
• GatewayManager
• FnmService
• HelperService
• UpdateService

For each service module, confirm that ReplicaStatus is Ready and HealthState is OK. In production deployments with multi-node Network Controllers, you can determine which node hosts the primary replica of each service and check the individual replica status using:

Get-NetworkControllerReplica

Verify that the Replica Status is Ready for all services across all nodes to ensure the Network Controller is functioning correctly.

Check MTU and Jumbo Frame Support on HNV Provider Logical Network¶

Insufficient MTU configuration or lack of Jumbo Frame support in the HNV Provider logical network is a common source of connectivity problems. The physical network ports and Ethernet cards must be configured with a sufficiently large MTU to accommodate the overhead introduced by VXLAN or NVGRE encapsulation. Without adequate MTU support, packets may be fragmented, leading to performance degradation or outright connection failures. This is particularly relevant in environments where Jumbo Frames are not enabled end-to-end.

Check Tenant VM Network Adapter Connectivity¶

Each virtual network adapter assigned to a tenant VM relies on a CA-PA mapping, linking the private Customer Address (CA) within the VM to the HNV Provider Address (PA) space in the SDN overlay network. These mappings are dynamically maintained within the OVSDB server tables on each Hyper-V host. Incorrect or missing mappings will prevent VMs from communicating within their virtual networks. Retrieve these mappings using the following PowerShell cmdlet:

# Get all PA-CA Mappings from the Hyper-V Host
Get-PACAMapping

Common Errors and Solutions Based on Configuration State¶

Certain error codes are frequently encountered in SDN environments, often indicating specific configuration issues or component failures. Understanding these common errors and their corresponding solutions can significantly expedite the troubleshooting process.

Error Code: HostUnreachable¶

Error Message:

MUX is Unhealthy (Common case is BGPRouter disconnected)

This error typically arises because the Credible Border Gateway Protocol (CBGP) peer, residing on the Routing and Remote Access Service (RRAS) (BGP VM) or a Top-of-Rack (ToR) switch, is unreachable or not successfully establishing peering. Solution: Thoroughly examine the BGP settings on both the Software Load Balancer Multiplexer resource and the BGP peer (ToR or RRAS VM). Verify IP addresses, Autonomous System Numbers (ASNs), and authentication settings to ensure correct BGP configuration and peering establishment.

Error Code: CertificateNotTrusted and CertificateNotAuthorized¶

Error Message:

Failed to connect to Mux due to network or cert errors

These errors point to certificate-related issues preventing secure communication. Solution: Analyze the numeric code accompanying the error message, as it corresponds to a winsock error code providing granular details about the certificate problem. Common certificate issues include:

• CertificateNotTrusted: The certificate presented by the remote endpoint is not trusted by the local system. This could be due to a missing or untrusted root certificate in the local certificate store. Ensure that the necessary certificates are properly installed and trusted on all relevant SDN components.
• CertificateNotAuthorized: The certificate is not authorized for the intended purpose or endpoint. Verify that the certificate is valid, correctly issued for the communicating entities, and configured for the appropriate services.

Error Code: HostNotConnectedToController¶

Error Message:

SLB host agent is not connected

This error indicates that the Software Load Balancer (SLB) Host Agent is unable to connect to the Network Controller, disrupting SLB functionality. Solution: Follow these steps to troubleshoot:

1. Verify SLB Host Agent Service Status: Confirm that the SLB Host Agent service is running on the Hyper-V host. Restart the service if it is stopped or in a failed state.
2. Examine SLB Host Agent Logs: Review the SLB host agent logs for detailed error messages and root cause analysis. These logs often provide valuable insights into connection failures, certificate rejections, or other issues preventing the agent from connecting to the SLBM (NC). Pay close attention to any certificate rejection errors, which might indicate certificate trust or authorization problems.

Error Code: DistributedRouterConfigurationFailure¶

Error Message:

Failed to configure the Distributed router settings on the host vNic

This error signifies a problem within the TCP/IP stack during the configuration of distributed router settings on a host virtual Network Interface Card (vNIC). Solution: This error often necessitates cleaning up residual Peer Authentication (PA) and Destination Rule (DR) Host Virtual Network Interface Cards (VNICs) on the server reporting the error. These VNICs might be in a corrupted or inconsistent state, preventing proper router configuration. Consult Microsoft documentation or support resources for detailed procedures on cleaning up PA and DR VNICs.

Error Code: PolicyConfigurationFailure¶

Error Messages:

• Failed to push vSwitch policies for a virtual machine network interface card (VmNic) due to certificate errors or connectivity error

• Failed to push Firewall policies for a VmNic due to certificate errors or connectivity errors

• Failed to push virtual network (VNet) policies for a VmNic due to certificate errors or connectivity errors

These errors commonly point to issues preventing the Network Controller from successfully pushing network policies to vSwitches or virtual machines. The root causes are often related to certificate problems or connectivity disruptions between the Network Controller and Hyper-V hosts. Solution:

• Certificate Verification: Ensure that the appropriate certificates have been deployed to all Hyper-V hosts and Network Controller nodes. Critically, the certificate subject name must precisely match the Fully Qualified Domain Name (FQDN) of the host. Certificate mismatches or invalid certificates will prevent secure policy distribution.
• Host Connectivity Check: Thoroughly verify network connectivity between the Hyper-V hosts and the Network Controller. Connectivity issues will obviously impede policy delivery. Refer back to the “Check Network Connectivity Between the Network Controller and Hyper-V Host” section to diagnose and resolve any connectivity problems.

Data Collection¶

Effective data collection is paramount for efficient problem resolution, especially when escalating issues to Microsoft support. Gathering relevant logs and diagnostic information beforehand significantly accelerates the support process and ensures quicker resolution.

Prerequisites¶

Before initiating data collection using the TSS (Troubleshooting Script Set) tool, ensure the following prerequisites are met:

1. Administrator Privileges: TSS must be executed by accounts possessing administrator privileges on the local system. This is essential for TSS to access system logs and perform diagnostic tasks effectively.
2. EULA Acceptance: The End-User License Agreement (EULA) must be accepted when running TSS for the first time. Once accepted, TSS will not prompt for EULA acceptance again on subsequent runs.
3. PowerShell Execution Policy: It is recommended to set the local machine’s PowerShell execution policy to RemoteSigned. This policy allows running scripts signed by a trusted publisher, enhancing security while enabling TSS to execute properly.

Steps for Data Collection¶

1. Download TSS: Download the TSS tool from https://aka.ms/getTSS on all nodes involved in the issue. Unzip the downloaded file to the C:\tss folder on each node.
2. Open Elevated PowerShell Prompt: Open an elevated PowerShell command prompt (Run as Administrator) and navigate to the C:\tss folder on each node.

3. Start Traces: Initiate the data collection process by executing the following cmdlet in the PowerShell prompt:

   TSS.ps1 -Scenario NET_SdnNC
   

   
   4. Accept EULA (if prompted): If this is the first time running TSS on the computer, accept the EULA when prompted.
   5. Allow Recording (PSR or Video): Allow TSS to record steps to reproduce the issue using Problem Steps Recorder (PSR) or video capture. This visual documentation is invaluable for understanding the issue context.
   6. Reproduce the Issue: Crucially, reproduce the SDN issue before proceeding. This ensures that the collected logs capture the relevant events leading up to and during the problem occurrence. Wait for the confirmation message on both client and server nodes (if collecting logs on both) before reproducing the issue.
   7. Finish Log Collection: After successfully reproducing the issue, enter Y in the PowerShell prompt to finalize the log collection process.

The collected traces will be compressed into a zip file and stored in the C:\MS_DATA folder on each node. These zip files can then be uploaded to a designated workspace for analysis by support personnel.

We encourage you to leverage this troubleshooting guide to resolve common SDN issues within your Windows Server environment. Should you encounter persistent problems, providing detailed information and collected data will significantly aid in efficient support and resolution. Feel free to share your experiences and questions in the comments below to foster a collaborative troubleshooting community.