Fix Code Coverage Issues: A Practical Guide for Visual Studio Users

Table of Contents

Visual Studio provides a powerful code coverage analysis tool designed to collect data for both native (C++) and managed (.NET) assemblies, typically in .dll or .exe file formats. This tool is invaluable for understanding how much of your codebase is being exercised by your tests, helping identify untested areas that may harbor hidden bugs. However, users sometimes encounter frustrating situations where the Code Coverage Results window displays errors such as “Empty results generated: ....” or shows no data at all.

This guide aims to be a comprehensive resource for troubleshooting and resolving the various common issues that can prevent the code coverage tool in Visual Studio from generating or displaying the expected results. By systematically addressing potential causes, you can ensure your code coverage analysis is accurate and useful.

What You Should See¶

When you initiate the code coverage analysis process, usually by selecting an Analyze Code Coverage command from the Test menu within Visual Studio, the tool begins its work. For the process to succeed, your project must build without errors, and importantly, your tests must execute successfully.

Upon successful completion of the build and test execution phases, you should see the Code Coverage Results window populated with data. This window typically displays a hierarchical view of your assemblies, namespaces, classes, and methods, indicating the percentage of code that was covered by the tests. You may need to expand the nodes within the window to view detailed coverage information at different levels of granularity.

Possible Reasons for Seeing No Results or Old Results¶

Several factors can lead to the code coverage tool failing to produce results or displaying outdated information. Identifying the specific cause is the first step towards a resolution.

You’re Not Using the Right Edition of Visual Studio¶

Analysis

The code coverage analysis feature, particularly the integrated tooling within Visual Studio, is not available in all editions. Attempting to run code coverage in an unsupported edition will result in the feature being unavailable or non-functional.

Explanation

Code coverage analysis, especially for professional development environments requiring integration with testing frameworks and build processes, is typically included in higher-tier editions of integrated development environments (IDEs). Microsoft’s licensing structure for Visual Studio places certain advanced diagnostic and testing tools, including comprehensive code coverage, into specific editions.

Resolution

Verify that you are using Visual Studio Enterprise edition. The code coverage tools are included as part of the testing and diagnostic suite available in this edition. If you are using a different edition (like Community or Professional), you will need to upgrade to Visual Studio Enterprise to utilize the integrated code coverage functionality.

No Tests Were Executed¶

Analysis

Inspect the Visual Studio Output window after attempting to run code coverage. Use the Show output from dropdown list and select Tests. Look for any warning messages or errors that indicate test execution failures or that no tests were discovered or run.

Explanation

The fundamental principle behind code coverage analysis in Visual Studio is that it monitors code execution during test runs. The tool instruments your code (adds hooks to track execution paths) and then observes which parts of the code are hit as your unit or integration tests execute. If no tests actually run, either because they failed to discover, encountered setup issues, or were explicitly filtered out, then no code execution will occur under the coverage tool’s observation. Consequently, there will be no data collected, resulting in empty coverage results.

Resolution

Before running code coverage, ensure that your tests can execute successfully on their own. Open the Test Explorer window (Test > Test Explorer). Try running all your tests by clicking Run All. Resolve any compilation errors, test failures, or infrastructure issues that prevent your tests from completing successfully. Only after confirming that your tests run without errors should you attempt to analyze code coverage.

You’re Looking at a Previous Result¶

Analysis

After running a new code coverage analysis, examine the Code Coverage Results window carefully. Check if the result set currently displayed is the most recent one generated from your last analysis run. Visual Studio can store multiple code coverage results from different runs.

Explanation

Visual Studio retains previous code coverage results, allowing you to compare coverage over time or between different test runs. When you run code coverage again, especially after making code or test changes, the window might not automatically switch to display the newest results. This can lead to confusion, as the displayed data, including source code colorization, might reflect an older state of your code or tests.

Resolution

Explicitly select the desired result set in the Code Coverage Results window. Typically, there’s a dropdown or list of available results. Make sure the currently selected result corresponds to the timestamp or identifier of your latest code coverage run. Running Analyze Code Coverage again will generate a new result set, and you must ensure this new set is active in the window to see the updated coverage data.

.pdb (Symbol) Files Are Unavailable¶

Analysis

Navigate to the output directory where your compiled assemblies (.dll or .exe files) are located. This is typically within your project’s bin\ folder, often under bin\Debug or bin\Release depending on your build configuration. Verify that for every assembly you expect to be covered, a corresponding .pdb (Program Database) file exists in the same directory.

Explanation

Program Database (.pdb) files are crucial symbol files generated during the compilation process. They contain vital mapping information between the executable code (Intermediate Language for .NET or native machine code) and the original source code lines. The code coverage tool relies heavily on .pdb files to understand the structure of your code, identify executable lines, and accurately attribute execution hits recorded during the test run back to your source files. If a .pdb file is missing, outdated, or does not match the corresponding assembly, the coverage tool cannot perform this mapping and will skip the assembly, resulting in it not appearing in the coverage results. The .pdb file must be generated from the exact same build as the .dll or .exe file being tested.

Resolution

Ensure that your project’s build settings are configured to generate .pdb files with full debug information. The specific settings vary slightly depending on the project type and Visual Studio version:

• For older .NET projects (using legacy .NET Framework build settings): Open the project properties, go to the Build tab, click the Advanced button, and check the Debug Info setting. It should be set to full or pdbonly.
• For modern .NET projects (targeting .NET Core or .NET 5+): Open the project properties, go to the Build tab, then the General section. Look for the Debug symbols setting and ensure it is set to Full or Portable.
• For C++ projects: Open the project properties, navigate to Linker > Debugging. Verify that the Generate Debug Info setting is configured to Generate Debug Information optimized for sharing and publishing (/DEBUG:FULL). This setting ensures that comprehensive debugging information, necessary for code coverage tools, is included in the .pdb file.

Additionally, confirm that the .pdb files are copied to the same output directory as their corresponding assemblies during the build process. If they are built to a different location (e.g., a separate symbols directory), you may need to adjust your build or test execution configuration to ensure the coverage tool can find them. Customizing code coverage analysis via a .runsettings file allows specifying symbol search paths if needed, but placing them alongside the assemblies is the most straightforward approach.

An Instrumented or Optimized Binary Is Used¶

Analysis

Determine if the assembly being tested has been previously processed by other profiling or optimization tools. Common culprits include Profile-Guided Optimization (PGO) applied during the build process or manual instrumentation using tools like vsinstr.exe or monitoring using vsperfmon.exe from the Visual Studio profiling tools.

Explanation

Code coverage tools work by injecting their own instrumentation into the target binaries to track execution. If a binary has already been modified (instrumented) or significantly altered (optimized) by another tool for a different purpose (like performance profiling), its structure may be fundamentally incompatible with the code coverage tool’s expectations. The code coverage engine detects this prior modification or optimization and, to avoid conflicts or incorrect data, will typically skip analyzing such assemblies entirely.

Resolution

Ensure that the binaries you are testing and analyzing for code coverage are standard builds without prior instrumentation or advanced optimization steps like PGO applied. Disable any build steps that perform manual instrumentation or apply profile-guided optimizations for the build configuration you are using for code coverage analysis. Use a fresh build of the non-instrumented/non-optimized binaries for your code coverage run.

Code Isn’t Managed (.NET) or Native (C++) Code¶

Analysis

Confirm the type of code base and the technologies involved in the assemblies and tests you are attempting to cover. Are you working exclusively with .NET (managed) code or standard C++ (native) code?

Explanation

The integrated code coverage analysis tool within Visual Studio is specifically designed to work with code compiled for the Common Language Runtime (CLR) in .NET (managed code) and standard compiled C++ code (native code). It utilizes specific instrumentation techniques applicable to these environments. If your project involves code written in other languages or compiled for different platforms or runtimes (e.g., JavaScript, Python, Java, or code running on specialized embedded platforms that don’t use the standard Windows/CLR execution model), the Visual Studio code coverage tool will not be able to analyze it. Even within a Visual Studio solution, if some projects use unsupported technologies, their code will not contribute to the coverage results.

Resolution

Understand the limitations of the Visual Studio code coverage tool. It is suitable for .NET and C++ projects. For other languages or platforms, you will need to seek out code coverage tools specifically designed for those environments. There is no built-in resolution within the standard Visual Studio tool for covering non-managed or non-native C++ code types.

Project Name Includes ‘DataCollector’¶

Analysis

Check the names of the projects within your solution that you expect to see coverage results for. See if any of these project names contain the string ‘DataCollector’.

Explanation

There is a known issue or convention where projects whose names contain the substring ‘DataCollector’ might be unintentionally excluded from code coverage analysis by the Visual Studio tool. This is likely due to internal logic within the testing or data collection framework that might try to avoid analyzing components perceived as part of the testing infrastructure itself.

Resolution

Rename the affected project(s) to remove the ‘DataCollector’ substring from their names. Choose a name that accurately reflects the project’s purpose but avoids this specific string. After renaming, rebuild the solution and attempt to run code coverage again.

Assembly Has Been Installed by NGen¶

Analysis

Determine if the assembly you are testing is being loaded from the Native Image Cache. This typically happens when the Ngen.exe (Native Image Generator) tool has been used to compile the .NET assembly into native code and install it into the GAC (Global Assembly Cache).

Explanation

Ngen.exe compiles .NET assemblies into native machine code to improve startup performance. When an application uses an assembly processed by NGen and installed in the GAC, the native image is loaded directly instead of the original Intermediate Language (IL) assembly being Just-In-Time (JIT) compiled. The Visual Studio code coverage tool primarily works by instrumenting the Intermediate Language (IL) code of managed assemblies before they are loaded or JIT-compiled. It is not designed to instrument or analyze code that has already been compiled to native images by NGen. Therefore, assemblies loaded from the native image cache will be skipped by the coverage tool.

Resolution

Ensure that the version of the assembly being loaded during your test run is the original MSIL version, not the native image generated by NGen. This often means configuring your test environment or application setup to load the assembly from its build output directory (bin\Debug or bin\Release) rather than relying on it being present and loaded from the GAC as a native image. Avoid processing assemblies with NGen if you intend to analyze them for code coverage.

The Custom .runsettings File Has Syntax Issues¶

Analysis

If you are using a custom .runsettings file to configure your test run and code coverage analysis, open the file and carefully inspect its structure and content. Look for any indications of malformed XML or incorrect syntax within the regular expressions used for including or excluding assemblies or types.

Explanation

Visual Studio allows you to customize various aspects of test execution and data collection, including code coverage settings, using a .runsettings file. This XML-based file can specify which assemblies, namespaces, types, or methods should be included in or excluded from the analysis. Errors in the XML structure (e.g., missing closing tags, incorrect element nesting) or syntax errors within the regular expressions used in the include/exclude rules can prevent the code coverage data collector from initializing or running correctly. If the .runsettings file is invalid, the tool may fail silently, produce empty results, or only show results from a previous, successful run.

Resolution

Thoroughly check your .runsettings file for errors:

1. XML Validation: Open the .runsettings file in an XML editor (Visual Studio’s built-in XML editor is suitable) which can often highlight syntax errors. Ensure the XML structure is well-formed and adheres to the expected schema for .runsettings files. Pay close attention to opening and closing tags and attribute syntax.
2. Regular Expression Validation: Each Include or Exclude rule within the CodeCoverage section uses regular expressions to match assembly names, namespaces, etc. Errors in regular expressions are common. Be particularly careful with special characters:
   • Parentheses ( and ) are used for grouping in regex. If you want to match a literal parenthesis in a name (e.g., a method signature like MyMethod(int)), you must escape them using a backslash: .*MyMethod\\(int\\).*.
   • Characters like * (zero or more of the preceding element) and + (one or more of the preceding element) have special meaning at the start of an expression. To match any string of characters, use .* (dot followed by asterisk), which matches any character (.) zero or more times (*).
   • Test your regular expressions using an online regex tester to ensure they match the intended assembly or type names correctly.

Correct any XML or regex errors in the .runsettings file, save it, and rerun the code coverage analysis.

Custom .runsettings File With Incorrect Exclusions¶

Analysis

If you are using a custom .runsettings file and are getting some results but missing coverage for specific assemblies or code sections that you expect to be covered, the issue might be overly broad or incorrect Exclude rules, or too restrictive Include rules.

Explanation

The Include and Exclude sections within the CodeCoverage node of a .runsettings file dictate which parts of your code base the coverage tool will analyze. An Exclude rule takes precedence over an Include rule if they both match. If your Exclude rules are too general, they might inadvertently filter out assemblies or code that you intended to include. Conversely, if you only specify Include rules, and they don’t precisely match the assemblies being loaded during the test run, those assemblies will be ignored.

Resolution

Troubleshoot your .runsettings include/exclude rules systematically:

1. Simplify: Temporarily remove all Include nodes from the .runsettings file and rerun code coverage. If results appear for more assemblies, the issue lies in your Include rules being too restrictive.
2. Simplify Further: If removing Include nodes doesn’t help or isn’t applicable, temporarily remove all Exclude nodes and rerun. If significantly more (or all) assemblies are now included, the problem is with your Exclude rules being too broad.
3. Refine: Once you’ve identified whether the issue is with includes or excludes, add them back in stages, testing after each addition.
   • When working with Include rules, ensure the regular expressions precisely match the full assembly names (including file extension .dll or .exe if specified in the rule, or just the base name depending on the regex pattern). Remember that assemblies are loaded from the build output directory, so the assembly name in the rule should match the file name found there.
   • When working with Exclude rules, make them as specific as possible. Avoid using very general patterns like .* unless you truly mean to exclude everything. Target specific assembly names, namespaces, or types that you know you want to omit (e.g., third-party libraries, auto-generated code).

4. Validate DataCollector Node: Ensure the <DataCollectors> node in your .runsettings file correctly specifies the Code Coverage data collector. It should look similar to this structure (refer to Microsoft documentation for the exact schema):

   <DataCollectors>
       <DataCollector friendlyName="Code Coverage" uri="vstest://Microsoft.CodeCoverage/v1" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceDataCollector, Version=..." />
   </DataCollectors>
   

By iteratively refining your include and exclude rules, you can pinpoint which specific rule is causing the unexpected filtering.

Some Code is Always Shown as Not Covered¶

Even when code coverage analysis runs successfully and produces results, you might notice that certain lines or blocks of code consistently appear as not covered, even though you believe they are being executed by your tests. This can happen for specific scenarios, particularly in native code.

Initialization Code in Native DLLs is Executed Before Instrumentation¶

Analysis

If you are working with native C++ code compiled into a DLL, and that DLL is statically linked to your test executable or another application that your tests invoke, you might observe that code within the DLL’s entry point function (DllMain) or code called directly from DllMain during the DLL loading process is marked as not covered.

Explanation

The Visual Studio code coverage tool works by performing instrumentation on binaries just before they are loaded and executed within the context of the test run. For dynamically loaded DLLs, the instrumentation happens effectively before any significant code within that DLL is executed. However, for statically linked or early-loaded native DLLs, the operating system loads and initializes the DLL as part of the main process startup. The DllMain function (or equivalent entry point) is invoked by the OS loader before the main application entry point is reached and potentially before the code coverage tool has fully attached and applied its instrumentation. Code executed during this very early initialization phase occurs outside the window of observation for the code coverage tool, thus appearing as not covered.

Resolution

This behavior is inherent to how native DLLs are loaded and initialized by the operating system and how the code coverage tool attaches. There is typically no straightforward resolution within the code coverage tool itself to mark this specific, early-executing initialization code as covered. If this affects only a small, non-critical portion of your code (like standard DLL entry point setup), it might be acceptable to live with this limitation. If critical logic resides in early initialization, consider refactoring the code so that it is called from later parts of the application or test setup, after the coverage tool has attached. However, this may not always be feasible or desirable depending on the code’s purpose.

Conclusion¶

Troubleshooting code coverage issues in Visual Studio can sometimes be challenging, but by systematically investigating the common causes outlined in this guide, you can often identify and resolve the problem. From ensuring you have the correct Visual Studio edition and that your tests are running, to verifying the presence and correctness of .pdb files and checking your .runsettings file for errors or incorrect rules, each step helps narrow down the potential source of empty or inaccurate results.

Remember that code coverage is a valuable metric, but it’s just one part of a comprehensive testing strategy. It tells you what code was executed, but not necessarily how well it was tested.

Did this guide help you fix your code coverage issues? Are there other scenarios you’ve encountered that aren’t covered here? Share your experiences or ask questions in the comments below! Your insights can help the community.