Home arr failed request tracing iis troubleshooting

Troubleshooting ARR with Failed Request Tracing: A Deep Dive into IIS

Updated: • 0 • min read
Table of Contents

Failed Request Tracing is an indispensable diagnostic feature integrated within Internet Information Services (IIS) on Windows Server 2016 and later. This powerful tool is specifically designed to capture detailed logs of request processing failures within the IIS web server. By enabling Failed Request Tracing, administrators and developers gain granular insight into the inner workings of IIS when errors occur, allowing for efficient identification and resolution of issues. This article will guide you through the process of configuring Failed Request Tracing rules to effectively debug failures and trace the request flow within Application Request Routing (ARR).

Goal

The primary objective is to learn how to set up Failed Request Tracing rules tailored for Application Request Routing and to understand the crucial information contained within the generated logs for effective troubleshooting. By the end of this guide, you will be equipped to diagnose and resolve issues related to request routing within your IIS environment using Failed Request Tracing.

Prerequisites

Before proceeding with the configuration and analysis of Failed Request Tracing for ARR, ensure the following prerequisites are met:

  • IIS on Windows Server 2016 or newer: You need a Windows Server 2016 (or a later version) operating system with the IIS role installed. Any SKU of Windows Server 2016 or newer will suffice.
  • Tracing Role Service for IIS: The Tracing role service must be installed within IIS. This component is essential for enabling and utilizing the Failed Request Tracing feature. You can install this role service through the Server Manager.
  • Microsoft Application Request Routing (ARR) and Dependent Modules: ARR must be installed and configured on your IIS server. ARR is a module that provides powerful URL-based routing, load balancing, and HTTP proxy functionalities for IIS. Ensure that ARR and its dependent modules are correctly installed. If ARR is not yet installed, it can be downloaded from the Microsoft Download Center. Follow the installation guide for detailed steps.
  • Minimum of Two Application Servers: You should have at least two application servers with functional websites and applications. These servers will serve as the backend servers for ARR to route requests to, allowing you to test and observe the routing behavior.
  • Basic ARR Configuration: It is assumed that you have already gone through the initial configuration of Application Request Routing. ARR should be in a working state before you start configuring Failed Request Tracing. This includes setting up server farms and basic routing rules.

IIS on Windows Server 2016

Step 1: Configure Failed Request Tracing Rules

Configuring Failed Request Tracing rules is the first crucial step to capturing relevant logs for ARR troubleshooting. You can configure these rules using either the IIS Manager UI, which provides a graphical interface, or through the command-line for more automated or scripting-based setups. Both methods achieve the same result, allowing you to choose the approach that best suits your workflow.

How to configure Failed Request Tracing rules using the UI

The IIS Manager UI provides a user-friendly way to configure Failed Request Tracing rules. Follow these steps:

  1. Launch IIS Manager: Open the Internet Information Services (IIS) Manager by searching for “inetmgr” in the Windows search bar or by running inetmgr in the command prompt.
  2. Select Default Web Site: In the Connections pane on the left side of the IIS Manager, select Default Web Site. This is where we will configure the tracing rules for the default website, which typically handles ARR routing.

  3. Open Failed Request Tracing Settings: In the Actions pane on the right side, under the Configure section, click on Failed Request Tracing…. This will open the “Edit Web Site Failed Request Tracing Settings” dialog box.

    Failed Request Tracing Settings
    4. Enable Failed Request Tracing: In the “Edit Web Site Failed Request Tracing Settings” dialog box, check the Enable checkbox. This activates Failed Request Tracing for the selected website.
    5. Save Changes: Click OK to save the changes and close the “Edit Web Site Failed Request Tracing Settings” dialog box.
    6. Open Failed Request Tracing Rules: With Default Web Site still selected in the Connections pane, double-click Failed Request Tracing Rules in the middle pane. This will open the Failed Request Tracing Rules feature page.
    7. Add New Rule: In the Actions pane on the right side, click Add…. This will launch the “Add Failed Request Tracing Rule Wizard”.
    8. Define Content to Trace: In the “Define Content to Trace” step of the wizard, select All content (*). This specifies that the tracing rule should apply to all content requested on the website. Then, click Next.
    9. Define Trace Conditions: In the “Define Trace Conditions” step, select Status code(s): and enter 200-399. This configures the rule to capture traces for requests that result in HTTP status codes within the 200-399 range, which typically indicates successful or redirect responses. While we are troubleshooting, tracing successful requests can also provide valuable insight into ARR’s routing decisions. Click Next.

    Trace Conditions
    10. Select Trace Providers: In the “Select Trace Providers” step, deselect ASP, ASPNET, and ISAPI Extension. Expand WWW Server. Under Areas:, deselect all options except for Rewrite and RequestRouting. ARR relies heavily on the URL Rewrite Module for inspecting incoming requests and making routing decisions. Therefore, enabling traces for both RequestRouting (ARR itself) and Rewrite (URL Rewrite Module) is crucial for comprehensive ARR troubleshooting.
    11. Finish Rule Creation: Click Finish to complete the wizard and create the Failed Request Tracing rule.

How to configure Failed Request Tracing rules using the command-line

For administrators who prefer command-line operations or need to automate the configuration process, Failed Request Tracing rules can be configured using appcmd.exe. Follow these steps:

  1. Open Command Prompt as Administrator: Open a command prompt with administrator privileges. This is essential for appcmd.exe to execute commands that modify IIS configuration.
  2. Navigate to inetsrv Directory: Navigate to the IIS installation directory by typing cd %windir%\system32\inetsrv and pressing Enter. This directory contains the appcmd.exe tool.

  3. Enable Failed Request Tracing: To enable Failed Request Tracing on the Default Web Site, run the following command:

    appcmd set site "Default Web Site" -traceFailedRequestsLogging.enabled:"true" /commit:apphost

    This command uses appcmd to set the traceFailedRequestsLogging.enabled property to “true” for the “Default Web Site”. The /commit:apphost parameter ensures that the changes are saved to the applicationHost.config file.

  4. Configure Failed Request Tracing Rules: To configure the Failed Request Tracing Rules to capture traces for status codes 200-399 and for Rewrite and RequestRouting areas, execute the following commands sequentially:

    appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /+"[path='*']"

    This command adds a new <traceFailedRequests> element for all paths (path='*').

    appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /+"[path='*'].traceAreas.[provider='WWW Server',areas='Rewrite,RequestRouting',verbosity='Verbose']"

    This command adds a <traceAreas> element within the <traceFailedRequests> element for the “WWW Server” provider, specifically targeting “Rewrite” and “RequestRouting” areas with “Verbose” verbosity. Verbose verbosity captures the most detailed logs, which are helpful for in-depth troubleshooting.

    appcmd.exe set config "Default Web Site" -section:system.webServer/tracing/traceFailedRequests /[path='*'].failureDefinitions.statusCodes:"200-399"

    This command sets the statusCodes attribute within the <failureDefinitions> element to “200-399” for the <traceFailedRequests> element, ensuring that traces are captured for requests with status codes in this range.

After executing these commands, Failed Request Tracing will be enabled and configured to capture logs based on the defined rules.

Command Prompt

Step 2: Analyze Failed Request Tracing Logs

Once Failed Request Tracing rules are configured, the next step is to generate traffic to your ARR setup and then analyze the Failed Request Tracing logs. These logs contain detailed information about the request processing pipeline, including ARR’s routing decisions and server interactions.

To view Failed Request Tracing logs

Follow these steps to locate and view the Failed Request Tracing logs:

  1. Navigate to Log Directory: Open File Explorer and navigate to the directory where Failed Request Tracing logs are written. The default location is %SystemDrive%\inetpub\Logs\FailedReqLogFiles\. You can directly paste this path into the File Explorer address bar.

  2. Identify Website Folder: Inside the FailedReqLogFiles directory, you will find folders named W3SVC1, W3SVC2, and so on. These folders correspond to different websites in IIS. To find the folder for the Default Web Site, select Default Web Site in IIS Manager, and then click Advanced Settings… in the Actions pane. In the “Advanced Settings” dialog box, note the value of the ID property. The folder name (e.g., W3SVC1) corresponds to this ID (e.g., ID 1).

    IIS Advanced Settings
    3. Clear Existing XML Files (Optional): If there are existing XML files in the website’s folder (e.g., W3SVC1), you might want to remove them to start with a clean set of logs. Open a command prompt, navigate to the website’s log folder (e.g., cd %SystemDrive%\inetpub\Logs\FailedReqLogFiles\W3SVC1), and type del *.xml to delete all XML files.
    4. Send Requests to ARR: Send requests to your Application Request Routing setup. These requests should be directed to the Default Web Site, which is configured to use ARR. If ARR is functioning correctly and routing requests to backend servers, you should receive 200-series responses. Since we configured tracing for status codes 200-399, these successful requests will also generate Failed Request Tracing logs.
    5. Verify Log Creation: List the files in the website’s log directory again. You should now see new XML files with names like fr[timestamp].xml. These are the Failed Request Tracing log files for the requests you just sent.
    6. Open and Analyze XML Log: Open one of the newly created XML files in a web browser (like Internet Explorer, Edge, Chrome, or Firefox). The XML file will be automatically styled using an XSL stylesheet (freb.xsl) to provide a readable and interactive view of the trace log.

    • Request Details: In the browser, click on Request Details.
    • Complete Request Trace: Select Complete Request Trace to view the entire trace log.
    • Expand All: Click Expand All to expand all sections of the trace log for a comprehensive view.

    You will see a detailed timeline of events that occurred during the processing of the request.

  3. Focus on Key Sections: Pay close attention to the following sections within the Failed Request Tracing log, as they are particularly relevant for ARR troubleshooting:

    • GENERAL_REQUEST_HEADERS:

      • Headers: This section displays the raw HTTP headers received by IIS for the incoming request. Reviewing these headers can help understand the client’s request and identify any issues related to headers being sent to ARR.

    • ARR_REQUEST_ROUTED:

      • WebFarm: Indicates the name of the server group (web farm) where ARR routed the request. This is crucial for verifying if ARR is routing to the intended server farm.
      • Server: Shows the specific destination server within the web farm to which the request was routed. This confirms the server selection made by ARR.
      • Algorithm: Specifies the load balancing algorithm used by ARR to select the destination server (e.g., Round Robin, Weighted Round Robin, Least Requests). This helps understand the load balancing mechanism in action.
      • RoutingReason: Provides the reason why a particular server was chosen for routing. This is invaluable for understanding ARR’s decision-making process and troubleshooting unexpected routing behavior.

    • ARR_SERVER_STATS:

      • State: Indicates the availability status of the destination server at the time of routing (e.g., Online, Offline). This helps diagnose routing issues related to server availability.
      • TotalRequests: Runtime statistic showing the total number of requests that have been sent to this server by ARR.
      • CurrentRequests: Runtime statistic showing the number of concurrent HTTP requests currently being processed by this server.
      • BytesSent: Runtime statistic indicating the amount of data (in KB) sent to this server by ARR.
      • BytesReceived: Runtime statistic indicating the amount of data (in KB) received from this server by ARR.
      • ResponseTime: Runtime statistic showing the average response time (in milliseconds) of this server. These statistics provide insights into the performance and health of the backend servers.

    • GENERAL_RESPONSE_HEADERS:

      • Headers: Displays the HTTP response headers sent back by the destination server to ARR. Examining these headers helps understand the response from the backend server and identify any issues in the response headers.

    • GENERAL_RESPONSE_ENTITY_BUFFER:

      • Buffer: Shows a portion of the response body (entity) received from the destination server. This can be useful for quickly inspecting the response content from the backend server.

    • Performance Timestamps (ARR_REQUEST_HEADERS_START, ARR_REQUEST_HEADERS_END, etc.):

      • These events with timestamps are added to profile the performance of ARR. They indicate the start and end times of various stages in ARR’s request processing, such as receiving request headers, sending request headers, receiving response headers, and receiving response entity. These timestamps can be used to analyze the time spent in different stages of ARR processing and identify potential performance bottlenecks.

Failed Request Tracing Log Example

If you are collecting Failed Request Tracing logs on a server core installation where a browser is not readily available, you can copy the log files (XML files and the freb.xsl stylesheet) to a computer with a browser to view and analyze them.

Summary

You have now successfully configured Failed Request Tracing rules specifically for Application Request Routing and learned how to analyze the generated logs. Failed Request Tracing is a powerful tool for gaining deep insights into ARR’s request processing, routing decisions, and interactions with backend servers. By examining the logs, you can effectively troubleshoot and debug various ARR-related issues, understand the load balancing algorithms in action, and optimize the performance of your application delivery infrastructure. This detailed tracing capability is essential for maintaining a robust and efficient IIS environment with Application Request Routing.

If you have any questions or further insights on using Failed Request Tracing with ARR, feel free to leave a comment below!

Post a Comment