Conquering 502 Errors: A Practical Guide for IIS Application Request Routing

Table of Contents

When working with Internet Information Services (IIS) Application Request Routing (ARR) deployments, encountering errors is part of the process. One of the common and often frustrating errors is “HTTP 502 - Bad Gateway”. This error signifies that ARR, acting as a proxy, was unable to successfully complete a request to an upstream server (a member server in the farm) and subsequently return a response to the client.

Understanding the nature of the 502 error in ARR is crucial for effective troubleshooting. A 502 error specifically points to a problem in the communication between the gateway (ARR) and the next server in the chain. It’s not an error originating from the client or the destination server directly delivering a typical HTTP error code (like 404 or 500) to the client through ARR. Instead, ARR itself reports the failure because its attempt to fulfill the client’s request via the upstream server failed at some point.

The reasons behind a 502 Bad Gateway error can vary. Common causes include the inability of ARR to establish a connection with the member server, the member server failing to respond in a timely manner, or the connection being terminated unexpectedly mid-request. Identifying the specific root cause is the critical first step in resolving the issue and requires systematic investigation into the request flow and server logs.

Understanding 502.3 Timeout Errors¶

One specific variation of the 502 error encountered with ARR is the “HTTP 502.3 - Bad Gateway” error. This particular substatus code often points towards a timeout condition. When ARR proxies a request to a backend server and does not receive a response within a predetermined time limit, it will terminate the request and return a 502.3 error to the client. This timeout can occur at various stages of the communication between ARR and the member server.

The detailed error page, if enabled on the ARR server, can provide valuable clues by including a specific error code. This error code is typically derived from the underlying WinHTTP library, which ARR uses to make outbound requests to the member servers. Decoding this WinHTTP error code can help pinpoint the exact nature of the timeout or connection issue.

Decoding Error Codes and Analyzing Logs¶

The error code displayed on the detailed error page is a key piece of information. Tools like err.exe (Microsoft Error Lookup Tool) can translate these numerical codes into more human-readable descriptions. For instance, a common error code associated with 502.3 is 12002, which when decoded using err.exe translates to ERROR_WINHTTP_TIMEOUT. This confirms that the issue is indeed related to a request timing out from ARR’s perspective.

You can also find this WinHTTP status code in the IIS logs on the ARR controller server. The sc-win32-status field in the IIS log entry for a 502.3 error will typically contain this code. Analyzing the IIS logs provides a historical record of requests and their outcomes, allowing you to identify the frequency and pattern of these errors.

Here’s an example of what a relevant IIS log entry snippet for a 502.3 timeout might look like, focusing on the key status fields:

sc-status	sc-substatus	sc-win32-status	time-taken
502	3	12002	29889

The sc-status is 502, the sc-substatus is 3, and the sc-win32-status is 12002, indicating the WinHTTP timeout. The time-taken field shows the total time elapsed for ARR to process the request before returning the error. In this example, the request took approximately 29.8 seconds before timing out. This specific time-taken value is often very close to the configured ARR proxy timeout setting, which defaults to 30 seconds.

Tracing Request Flow with Failed Request Tracing¶

To gain deeper insight into what happened with a specific request, Failed Request Tracing (FREB) in IIS is an invaluable tool. By enabling FREB for 502 errors on the ARR server, you can capture detailed events for each failing request. This trace log shows the internal processing steps ARR takes, including when and where it attempted to route the request.

Within the FREB log on the ARR server, the ARR_SERVER_ROUTED event is particularly useful. This event indicates which backend server ARR selected for the request and provides information like the server’s IP address and internal status metrics. Crucially, it also logs an X-ARR-LOG-ID header that is added to the forwarded request.

77. ARR_SERVER_ROUTED  RoutingReason="LoadBalancing", Server="192.168.0.216", State="Active", TotalRequests="3", FailedRequests="2", CurrentRequests="1", BytesSent="648", BytesReceived="0", ResponseTime="15225" 16:50:21.033
82. GENERAL_SET_REQUEST_HEADER HeaderName="X-ARR-LOG-ID", HeaderValue="dbf06c50-adb0-4141-8c04-20bc2f193a61", Replace="true" 16:50:21.033

The X-ARR-LOG-ID is a unique identifier for that specific request instance within the ARR system. This ID is passed to the member server. By enabling FREB on the member server as well and filtering by this X-ARR-LOG-ID, you can correlate the request on the backend server and see how it was processed there. This allows you to trace the request’s journey end-to-end.

Examining the FREB log on the member server using the X-ARR-LOG-ID can reveal what the member server was doing with the request and whether it completed successfully from its perspective. For example, the member server’s FREB trace might show the request being processed and a response being generated.

185. GENERAL_REQUEST_HEADERS Headers="Connection: Keep-Alive Content-Length: 0 Accept: */* Accept-Encoding: gzip, deflate Accept-Language: en-US Host: test Max-Forwards: 10 User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0) X-Original-URL: /time/ X-Forwarded-For: 192.168.0.204:49247 X-ARR-LOG-ID: dbf06c50-adb0-4141-8c04-20bc2f193a61
... (multiple entries skipped for brevity)
345. GENERAL_FLUSH_RESPONSE_END BytesSent="0", ErrorCode="An operation was attempted on a nonexistent network connection. (0x800704cd)" 16:51:06.240

In this trace snippet from the member server, the X-ARR-LOG-ID matches the one from the ARR server trace, confirming it’s the same request. The GENERAL_FLUSH_RESPONSE_END event indicates the server finished processing and attempted to send data. The ErrorCode 0x800704cd translates to ERROR_NETNAME_DELETED, suggesting the connection was terminated unexpectedly by the client (ARR, in this case) while the server was still processing or sending the response.

Correlating the timestamps between the ARR server’s timeout (e.g., ~30 seconds in the time-taken IIS log field) and the member server’s processing time (e.g., 45 seconds in its IIS log time-taken field) further clarifies the scenario. If the member server took longer to process the request than the ARR proxy timeout setting, ARR would inevitably time out and close the connection, leading to the 502.3 error with ERROR_WINHTTP_TIMEOUT on the ARR side and potentially ERROR_NETNAME_DELETED on the member server side as it tried to write to a closed connection.

Identifying the Root Cause of Timeouts¶

Based on the analysis of IIS logs and Failed Request Tracing, if the time-taken on the ARR server closely matches the ARR proxy timeout setting (default 30 seconds), while the time-taken on the member server is longer, the primary cause is simple: the ARR timeout is shorter than the backend application’s execution time.

In the Server Farm configuration in IIS Manager, under Proxy settings, you can find the “Timeout (in seconds)” setting.

If the application is genuinely expected to take longer than 30 seconds for certain requests (e.g., complex reports, large data processing), increasing this timeout value in ARR is the direct solution. However, simply increasing the timeout might just mask an underlying performance problem on the backend server. It’s essential to investigate why the request is taking so long on the member server. This could involve profiling the application code, analyzing database queries, checking external service dependencies, or looking for resource bottlenecks (CPU, memory, disk I/O, network) on the member server.

Besides the overall proxy timeout, other WinHTTP timeouts can lead to ERROR_WINHTTP_TIMEOUT (code 12002). These are less common in typical ARR setups but are worth understanding:

• ResolveTimeout: Occurs if DNS name resolution for the member server takes too long.
• ConnectTimeout: Occurs if establishing a TCP connection to the member server after name resolution takes too long.
• SendTimeout: Occurs if sending the request body to the member server takes too long.
• ReceiveTimeout: Occurs if receiving the entire response from the member server takes too long after headers have been received.

If the failure happens during the ResolveTimeout or ConnectTimeout phases, you might not see any activity logged on the member server’s IIS or HTTPERR logs, as the connection was never fully established. In such cases, network tracing or WinHTTP tracing (discussed later) becomes essential to diagnose why the initial connection failed or timed out. Network connectivity issues, firewall blocks, or incorrect server addresses can cause these specific timeouts.

Troubleshooting 502.3 Connection Termination Errors¶

Apart from simple timeouts where the backend server is just slow, a 502.3 error can also result from the connection between ARR and the member server being prematurely terminated. This often happens when the backend application code explicitly closes the connection or encounters an unhandled error that causes the worker process to crash or the request to abort.

Consider an application on the member server that calls Response.Close() or similar functions before the response is fully sent or in the middle of sending data. When ARR is proxying this request, it expects a complete HTTP response. If the connection is abruptly closed by the member server, ARR detects this as a failure to complete the proxied request and returns a 502.3 error.

If you reproduce this scenario and have detailed errors enabled, you might see an error code like 0x80072efe. Using err.exe, this translates to ERROR_INTERNET_CONNECTION_ABORTED. This code clearly indicates that the connection was terminated unexpectedly by the remote party (the member server).

Failed Request Tracing on the ARR server would show the 502.3 error. FREB on the member server might show the request being processed up to the point where the connection was terminated. However, unlike the timeout scenario, you might not find a corresponding successful entry (HTTP 200) in the member server’s IIS logs if the request didn’t complete normally. Instead, the member server might log the event in its HTTPERR log, which records low-level HTTP API errors.

An entry in the HTTPERR log might look like this:

HTTP/1.1 GET /time/ - 1 Connection_Dropped DefaultAppPool

This log entry indicates a request for /time/ received via HTTP/1.1 resulted in a Connection_Dropped error. This confirms that the member server dropped the connection unexpectedly. The - 1 field before Connection_Dropped usually indicates the socket error, which could be 0 or other Winsock error codes depending on the exact timing and nature of the drop.

Investigating the application code running on the member server is crucial when you see ERROR_INTERNET_CONNECTION_ABORTED or Connection_Dropped in HTTPERR. Look for code that manipulates the response stream or the connection directly, such as Response.Close(), Response.End(), or unhandled exceptions that crash the process handling the request. Debugging the application on the member server is often necessary to find the exact line of code causing the connection termination.

Another scenario for connection termination leading to a 502.3 error is when the member server sends an invalid or incomplete response. If the member server starts sending response headers or body but then crashes, hangs, or explicitly closes the connection (e.g., by calling Response.Close() after partial output followed by Response.Flush()), ARR will detect this as an incomplete or malformed response and return a 502.3 error. An example error code for this could be 0x80072f78, which corresponds to ERROR_INTERNET_UNEXPECTED_SOCKET_ERROR. This suggests an issue occurred while ARR was trying to read data from the socket connected to the member server.

In such cases, network traces are invaluable. Capturing a network trace simultaneously on both the ARR server and the problematic member server allows you to see the actual packets exchanged. You can observe the HTTP conversation and identify exactly when and why the connection was closed or if malformed data was sent. For traffic over SSL/TLS, standard network sniffers won’t decrypt the payload, necessitating the use of WinHTTP/WebIO tracing for insight into the encrypted communication layer.

Diagnosing 502.4 Errors (No Available Server)¶

The HTTP 502.4 error, often accompanied by a win32 status code of 0 (0x00000000), indicates that Application Request Routing could not find an appropriate backend server to route the request to. This is different from 502.3 where a server was found but failed during communication. 502.4 means ARR couldn’t even initiate the request towards a backend server because none were deemed available or healthy.

This typically occurs when all member servers in the configured server farm are marked as unhealthy or offline by ARR’s health monitoring mechanisms. If ARR’s health checks fail for all registered servers, it has nowhere to send incoming client requests and thus returns a 502.4 error.

Checking Server Availability and Common Issues¶

The first step in troubleshooting a 502.4 error is to check the status of the member servers within the ARR server farm configuration in IIS Manager. Navigate to the Server Farm node, select your farm, and then click on the “Servers” feature.

This view lists all servers in the farm and their current status (e.g., Active, Offline). If all servers show as Offline, this is the reason for the 502.4 error. To attempt to bring a server back online, right-click on it and select “Add to Load Balancing”.

If servers are marked as Offline and you cannot bring them back online, investigate why ARR’s health checks are failing or why ARR cannot reach them. Potential causes include:

• Network Connectivity Issues: Firewalls blocking traffic between ARR and the member servers, incorrect routing, or network outages. Use ping or telnet from the ARR server to the member server’s IP and health check port (usually 80 or 443) to verify basic network reachability.
• Health Check URL Problems: ARR uses a health check URL to determine server health. If this URL is misconfigured, returns an error code (like 404, 500), takes too long to respond, or the application pool serving it is stopped, ARR will mark the server as unhealthy. Verify the health check URL is correct and accessible directly from the member server and from the ARR server.
• Application Pool Issues: The application pool running the health check or the main application on the member server might be stopped, crashed, or unresponsive. Check the Application Pools in IIS Manager on the member server and look at Windows Event Logs for crash information.
• Server Resource Exhaustion: The member server might be overloaded (high CPU, memory, or network usage) preventing it from responding to health checks or requests.
• Web Farm Framework (WFF) Issues: If using WFF, a restart of an application pool managed by WFF can sometimes cause servers to be marked offline. Restarting the Web Farm Service might resolve this.

The “Trace Messages” pane within the Server Farm’s “Servers” feature can sometimes provide specific error messages related to health check failures, offering more detailed clues. Analyzing these trace messages should be a standard step when troubleshooting 502.4 errors.

Advanced Troubleshooting with WinHTTP/WebIO Tracing¶

While network packet capture tools like WireShark or Microsoft Network Monitor (Netmon) are excellent for seeing traffic flow, they are less effective when the communication between ARR and the member servers is encrypted with SSL/TLS. In such cases, you need a way to inspect the data before it’s encrypted or after it’s decrypted by the operating system’s HTTP stack. This is where WinHTTP (or WebIO on newer OS versions) tracing becomes invaluable.

WinHTTP/WebIO tracing logs the activity of the WinHTTP API, which ARR uses for its outbound requests. This trace can show the details of connection attempts, data sending/receiving, timeouts, and errors, even for SSL traffic.

Capturing the Trace¶

On Windows Server 2008 R2 and later (including Windows 7+), you can use the netsh trace command-line utility to capture WinHTTP/WebIO events. You’ll need to run this command from an administrative command prompt on the ARR server where the 502 error originates.

First, create a temporary directory if you don’t have one, e.g., C:\temp.

mkdir c:\temp

Then, start the trace:

netsh trace start scenario=internetclient capture=yes persistent=no level=verbose tracefile=c:\temp\net.etl

The scenario=internetclient captures events related to WinHTTP/WebIO used by applications like ARR. capture=yes includes packet capture details where possible. persistent=no means the trace stops when you issue the stop command. level=verbose provides detailed event information. tracefile specifies the output file path.

Once the trace is running, reproduce the 502 error you are troubleshooting. Make the request that you know results in the error.

After reproducing the error, stop the trace using the following command in the administrative command prompt:

netsh trace stop

The stop command takes a few moments to process the captured data and write it to the .etl file. You should find a net.etl file and potentially a net.cab file in the C:\temp directory. The .cab file contains system information that can aid analysis.

Analyzing the Trace with Netmon¶

The .etl trace file captured by netsh trace needs to be analyzed using a tool capable of parsing Event Tracing for Windows (ETW) data, specifically the Microsoft-Windows-WinHttp or Microsoft-Windows-WebIO providers. Microsoft Network Monitor 3.4 or later is a suitable tool for this.

1. Install and Open Netmon 3.4+: Download and install Netmon if you haven’t already.
2. Open the Trace File: Go to File > Open > Capture and select the net.etl file you captured.
3. Set Parser Profile: Ensure Netmon is using the correct parsers. Go to Tools > Options, select the “Parser Profiles” tab, choose the “Windows” profile, and click “Set as Active”. This loads parsers for Windows-specific events, including WinHTTP/WebIO.
4. Filter by Process: ARR runs within a w3wp.exe worker process. To focus the trace, identify the PID of the w3wp.exe process hosting your ARR site/application pool. You can find this in IIS Manager under Worker Processes. Once you have the PID, right-click on any event from that w3wp.exe in the trace and select “Add UT Process name to display filter”. This will add a filter like UTProcessName == "w3wp.exe (<PID>)".
5. Filter by Protocol: Further refine the filter to only show WinHTTP/WebIO events by adding AND ProtocolName == "WINHTTP_MicrosoftWindowsWinHttp" (or similar for WebIO on newer OS). Your filter might look like:

UTProcessName == "w3wp.exe (<Your_PID>)" AND ProtocolName == "WINHTTP_MicrosoftWindowsWinHttp"

Replace <Your_PID> with the actual process ID of your ARR worker process. Apply the filter.

Interpreting Trace Examples¶

With the filter applied, scroll through the trace events chronologically. Look for events indicating errors, timeouts, or connection issues around the time you reproduced the 502 error.

Example 1: Timeout due to backend taking longer than ARR timeout

Look for events showing attempts to receive a response (::sys-recver processing WebReceiveHttpResponse completion) followed by error codes. A common pattern for the scenario where the backend is slow and ARR times out is:

...
WINHTTP_MicrosoftWindowsWinHttp::sys-recver processing WebReceiveHttpResponse completion (error-cdoe = ? (0x5b4), overlapped = ...)
WINHTTP_MicrosoftWindowsWinHttp::sys-recver failed to receive headers; error = ? (1460)
WINHTTP_MicrosoftWindowsWinHttp::ERROR_WINHTTP_FROM_WIN32 mapped (?) 1460 to (ERROR_WINHTTP_TIMEOUT) 12002
WINHTTP_MicrosoftWindowsWinHttp::sys-recver returning ERROR_WINHTTP_TIMEOUT (12002) from RecvResponse()
...

The error code 1460 is WSAETIMEDOUT (socket timeout), which WinHTTP maps to ERROR_WINHTTP_TIMEOUT (12002). The context here indicates the timeout occurred while waiting to receive the response from the backend server. The timestamps of these events will be roughly the ARR proxy timeout value after the request was sent.

Example 2: Timeout due to inability to connect

If the error is a ConnectTimeout (e.g., backend server is down or unreachable), the trace might show timeout errors during the connection phase:

...
WINHTTP_MicrosoftWindowsWinHttp::sys-recver processing WebReceiveHttpResponse completion (error-cdoe = WSAETIMEDOUT (0x274c), overlapped = ...)
WINHTTP_MicrosoftWindowsWinHttp::sys-recver failed to receive headers; error = WSAETIMEDOUT (10060)
WINHTTP_MicrosoftWindowsWinHttp::ERROR_WINHTTP_FROM_WIN32 mapped (WSAETIMEDOUT) 10060 to (ERROR_WINHTTP_TIMEOUT) 12002
WINHTTP_MicrosoftWindowsWinHttp::sys-recver returning ERROR_WINHTTP_TIMEOUT (12002) from RecvResponse()
...

Here, the initial error code 10060 (WSAETIMEDOUT) directly points to a socket connection timeout. This indicates ARR couldn’t establish a connection within the allowed time. This is typical if the backend server is offline, a firewall is blocking the connection, or network latency is excessively high.

WinHTTP/WebIO tracing provides a low-level view of the communication attempts, offering crucial context, especially when network traces are obscured by SSL/TLS or when standard application logs aren’t providing sufficient detail about the connection phase.

Preventing Future 502 Errors¶

Proactive measures can significantly reduce the occurrence of 502 errors in ARR deployments. Monitoring, performance tuning, and robust health checks are key strategies.

• Monitor Backend Performance: Implement monitoring on your backend member servers. Track key performance counters (CPU, memory, request queue, application-specific metrics) and application response times. Alerting on high resource utilization or slow responses allows you to address performance bottlenecks before they cause timeouts at the ARR layer.
• Optimize Backend Applications: Continuously profile and optimize the code running on your member servers. Long-running requests are a primary source of timeouts. Identify and fix inefficient database queries, slow external API calls, or complex processing that holds up requests.
• Tune ARR Timeouts Prudently: While increasing the ARR proxy timeout can resolve issues with legitimately long-running processes, avoid setting it excessively high. A very long timeout can tie up ARR resources and potentially hide underlying performance problems. Set the timeout based on the expected maximum processing time for your longest legitimate requests.
• Robust Health Checks: Configure comprehensive health checks in your ARR server farm. Instead of a simple static page, use a health check URL that exercises critical components of your application (e.g., database connection, external service availability). This allows ARR to quickly detect unhealthy servers and take them out of rotation, preventing 502.4 errors and routing traffic only to healthy instances.
• Load Balancing Configuration: Review your ARR load balancing algorithms and settings. Ensure server weights and load distribution methods are appropriate for your workload.
• Network Reliability: Ensure the network path between your ARR servers and member servers is stable and low-latency. Network issues can directly cause connection timeouts or terminations.

By combining monitoring, performance optimization, careful configuration, and thorough diagnostics when errors occur, you can build a more resilient and reliable ARR environment. Understanding the different types of 502 errors and having a systematic troubleshooting approach, including leveraging tools like IIS logs, Failed Request Tracing, and WinHTTP tracing, empowers you to quickly identify and resolve issues.

```mermaid
graph TD
A[Client Request] → B{ARR};
B – Proxies Request → C[Member Server];
C – Processes Request → D{Response};
D – Sends Response → B;
B – Sends Response → A;

B -- Timeout Occurs (e.g., waiting for C) --> E[502.3 Timeout];
C -- Connection Closed Abruptly --> F[502.3 Connection Terminated];
B -- No Healthy Server Found --> G[502.4 No Server];

style E fill:#f9f,stroke:#333,stroke-width:2px
style F fill:#f9f,stroke:#333,stroke-width:2px
style G fill:#f9f,stroke:#333,stroke-width:2px

```

This diagram illustrates the general flow and where the different types of 502 errors discussed can manifest. A request arrives at ARR, is proxied to a member server. Failures at the ‘Timeout Occurs’ or ‘Connection Closed Abruptly’ points lead to 502.3, while inability to even send the request to C leads to 502.4.

Essential Tools for Troubleshooting¶

Throughout this guide, several tools have been mentioned as critical for diagnosing 502 errors:

• IIS Logs: Provide historical records of requests and their outcomes, including sc-status, sc-substatus, and sc-win32-status.
• HTTPERR Logs: Capture low-level errors detected by the HTTP.sys kernel-mode driver on both ARR and member servers, useful for connection-level issues.
• Failed Request Tracing (FREB): Detailed request processing logs within IIS, showing events at various pipeline stages, crucial for correlating activity between ARR and member servers using X-ARR-LOG-ID.
• Err.exe (Error Lookup Tool): Translates Windows and WinHTTP error codes into descriptive text.
• Network Monitor / WireShark: Packet sniffers to analyze network traffic between servers. Essential for diagnosing connectivity problems but less useful for encrypted traffic payload analysis without additional steps.
• Netsh Trace (for WinHTTP/WebIO): Built-in Windows tool to capture detailed traces of the WinHTTP/WebIO API, invaluable for diagnosing issues with encrypted connections.
• Performance Monitor (Perfmon): Tracks system resources (CPU, memory, disk, network) and IIS/ASP.NET-specific counters to identify performance bottlenecks on member servers.

Mastering the use of these tools will greatly enhance your ability to diagnose and resolve 502 errors in your IIS ARR environment. Each tool provides a different perspective on the problem, and combining insights from multiple sources often leads to a quicker resolution.

Have you encountered specific 502 errors with IIS ARR that were particularly challenging to solve? Share your experiences or questions in the comments below!