Operations Manager Agent Grayed Out? Troubleshoot and Resolve Connectivity Issues

Table of Contents

This article provides comprehensive guidance on how to troubleshoot and resolve issues where an agent, management server, or gateway appears unavailable or grayed out within System Center Operations Manager (OpsMgr). Understanding the different states of these components is crucial for effective monitoring and management. We will explore common causes, strategic troubleshooting approaches, and detailed resolution steps for various scenarios.

In Operations Manager, the state of an agent, management server, or gateway is indicated by the color of its name and icon in the Monitoring pane. These visual cues provide immediate insight into the health and connectivity status of your monitored infrastructure. It is essential to recognize these states to quickly identify and address potential problems.

State	Appearance	Description
Healthy	Green check mark	The agent or management server is functioning as expected and running normally.
Critical	Red check mark	Indicates a significant problem or issue detected on the agent or management server that requires immediate attention.
Unknown	Gray agent name, gray check mark	This state signifies that the health service watcher on the management server, responsible for monitoring the health service on the managed computer, is no longer receiving heartbeats from the agent. Previously, heartbeats were received, and the agent was reported as healthy. This condition implies that management servers are no longer receiving any data or information from the affected agent. It often occurs if the computer running the agent is powered off or experiencing severe network or connectivity issues.
Unknown	Green circle, no check mark	This state indicates that the status of a discovered item is unknown. It typically means that no specific monitor has been configured or is available for this particular discovered item.

Causes of a Gray State¶

An agent, a management server, or a gateway can become unavailable and display a gray state for several underlying reasons. Identifying the root cause is the first step towards effective resolution. These issues can range from simple connectivity problems to more complex performance or configuration errors within the Operations Manager environment.

Common causes include:

• Heartbeat Failure: The primary mechanism for agents to signal their active status to management servers. A failure here directly results in a gray state.
• Invalid Configuration: Incorrect settings for agents, management servers, or management packs can prevent proper communication and operation.
• System Workflows Failure: Essential internal processes within Operations Manager may fail, impacting the health service and agent reporting.
• Operations Manager Database or Data Warehouse Performance Issues: Bottlenecks or performance problems with the underlying SQL databases can cause delays and communication failures.
• Management Server or Gateway Server Performance Issues: Overloaded or under-resourced management or gateway servers may struggle to process agent data.
• Network or Authentication Issues: Connectivity problems, firewall blocks, DNS resolution failures, or incorrect authentication settings can sever communication paths.
• Health Service is Not Running: The core service responsible for monitoring and communication on the agent, management server, or gateway may have stopped.

Issue Scope¶

Before embarking on troubleshooting a grayed-out agent issue, it is vital to first understand your Operations Manager topology. Defining the scope of the problem will help you pinpoint the affected areas and prioritize your troubleshooting efforts. Consider asking the following questions to gather essential context:

• How many agents are currently affected by this issue?
• Are the agents experiencing the problem located within the same network segment, or are they geographically dispersed?
• Do all the affected agents report to the same specific management server or gateway, or are multiple parent servers involved?
• How frequently do the agents enter and remain in a gray state, and is there a discernible pattern to their unavailability?
• What methods do you typically use to recover from this situation, such as restarting the agent health service, clearing the agent cache, or do you rely on automatic recovery mechanisms?
• Are heartbeat failure alerts being generated specifically for these agents, indicating a fundamental communication breakdown?
• Does this issue tend to occur during a particular time of day, suggesting a correlation with scheduled tasks, backups, or peak network activity?
• Does the problem persist if you manually fail over these agents to a different management server or gateway, testing alternative communication paths?
• When did this problem initially begin, helping to identify any recent changes in your environment?
• Were any significant changes recently made to the agents themselves, the management servers, the gateway, or the overall management group configuration?
• Are the affected agents part of Windows clustered systems, which may introduce additional complexities?
• Is the Health Service State folder on the affected machines properly excluded from antivirus scanning, as this is a common cause of performance degradation and instability?

Troubleshooting Strategy¶

Your troubleshooting strategy will largely be determined by which Operations Manager component is inactive and its position within your overall topology. The widespread nature of the problem will also guide your approach. A methodical investigation ensures that you start at the most logical point to diagnose the issue effectively.

Consider the following conditions to effectively strategize your troubleshooting:

• If only a subset of agents reporting to a particular management server or gateway is unavailable, your troubleshooting efforts should logically begin at the level of that specific management server or gateway. This narrows the focus to potential issues with the component directly responsible for those agents.
• Similarly, if several gateways that report to a specific management server become unavailable, the troubleshooting process should commence at that particular management server level. This indicates a potential issue with the server coordinating those gateways.
• For agentless systems, network devices, and Unix/Linux servers, troubleshooting should always start at the agent, management server, or gateway that is actively monitoring these objects. These monitoring components are the direct point of contact and data collection for the grayed-out items.
• Generally, troubleshooting should always start at the level immediately above the unavailable component in your Operations Manager hierarchy. This hierarchical approach helps in systematically isolating the problem and identifying the failure point.

Scenario 1: Intermittent Gray State with Cache Resolution¶

In this scenario, only a few agents are affected by the gray state issue. These agents report to different management servers, suggesting the problem is localized rather than server-wide. Agents enter an unavailable state on a regular basis, but you find that clearing the agent cache temporarily resolves the issue. However, the problem consistently recurs after a few days, indicating an underlying, persistent cause.

Resolution for Scenario 1¶

To effectively resolve the issue described in this scenario, follow these steps methodically. These actions aim to address common causes of intermittent gray states and improve agent stability.

1. Apply Appropriate Hotfixes: Ensure that the operating systems of the affected agents have all relevant and recommended hotfixes applied. Keeping systems updated can resolve known bugs that contribute to health service instability or communication issues.
2. Exclude Agent Cache from Antivirus Scanning: Configure your antivirus software to exclude the Operations Manager agent cache directory from real-time scanning. Antivirus interference is a frequent cause of performance degradation and unexpected behavior for the health service. Refer to official Microsoft documentation for specific recommendations on antivirus exclusions related to Operations Manager.
3. Stop the Health Service: On each affected agent, stop the System Center Management service (also known as the Health Service). This ensures that no files are in use while you perform the next step.
4. Clear the Agent Cache: Manually delete the contents of the agent’s Health Service State folder. This effectively clears any corrupted or stale configuration and operational data that might be causing the issue. The health service will rebuild this cache upon restart.
5. Start the Health Service: After clearing the cache, restart the System Center Management service. Monitor the agent’s state closely to confirm if it returns to a healthy status and remains stable over time.

Scenario 2: Constant Gray State with Cache Ineffectiveness¶

This scenario involves only a few agents displaying a gray state, and these agents report to various management servers. Crucially, these agents remain constantly inactive, and attempts to clear the agent cache do not provide a resolution. This suggests a more severe or fundamental problem that is not related to transient cache corruption.

Resolution for Scenario 2¶

To address agents that remain constantly grayed out, even after clearing the cache, a deeper investigation into the health service and event logs is required. Follow these steps to diagnose and resolve the issue:

1. Verify Health Service Status: Determine whether the Health Service is currently turned on and running on the affected agent, management server, or gateway. If the health service has stopped responding, consider generating an ADPlus dump in service hang mode. This can help to capture crucial diagnostic information to pinpoint the cause of the problem, such as deadlocks or unresponsive threads.

2. Examine Operations Manager Event Log: Carefully review the Operations Manager event log on the agent to identify any of the following specific events. These event IDs often provide valuable clues about workflow failures, Run As account issues, or connectivity problems. Understanding the messages associated with these events is critical for targeted troubleshooting.

   • Event ID: 1102 (HealthService): “Rule/Monitor ‘%4’ running for instance ‘%3’ with id:’%2’ cannot be initialized and will not be loaded. Management group ‘%1’”. Indicates a workflow loading failure.
   • Event ID: 1103 (HealthService): “Summary: %2 rule(s)/monitor(s) failed and got unloaded, %3 of them reached the failure limit that prevents automatic reload. Management group ‘%1’”. A summary of multiple workflow failures.
   • Event ID: 1104 (HealthService): “RunAs profile in workflow ‘%4’, running for instance ‘%3’ with id:’%2’ cannot be resolved. Workflow will not be loaded. Management group ‘%1’”. Implies issues with Run As account resolution.
   • Event ID: 1105 (HealthService): “Type mismatch for RunAs profile in workflow ‘%4’, running for instance ‘%3’ with id:’%2’. Workflow will not be loaded. Management group ‘%1’”. Indicates a mismatch in the Run As profile configuration.
   • Event ID: 1106 (HealthService): “Cannot access plain text RunAs profile in workflow ‘%4’, running for instance ‘%3’ with id:’%2’. Workflow will not be loaded. Management group ‘%1’”. Access denied to plain text Run As profiles.
   • Event ID: 1107 (HealthService): “Account for RunAs profile in workflow ‘%4’, running for instance ‘%3’ with id:’%2’ is not defined. Workflow will not be loaded. Please associate an account with the profile. Management group ‘%1’”. Run As account is not defined or associated.
   • Event ID: 1108 (HealthService): “An Account specified in the Run As Profile ‘%7’ cannot be resolved. Specifically, the account is used in the Secure Reference Override ‘%6’. This condition may have occurred because the Account is not configured to be distributed to this computer. To resolve this problem, you need to open the Run As Profile specified below, locate the Account entry as specified by its SSID, and either choose to distribute the Account to this computer if appropriate, or change the setting in the Profile so that the target object does not use the specified Account. Management Group: %1 Run As Profile: %7 SecureReferenceOverride name: %6 SecureReferenceOverride ID: %4 Object name: %3 Object ID: %2 Account SSID: %5”. A detailed message about Run As account distribution issues.
   • Event ID: 4000 (HealthService): “A monitoring host is unresponsive or has crashed. The status code for the host failure was %1.” Indicates that a Monitoringhost.exe process has crashed.
   • Event ID: 21016 (OpsMgr Connector): “OpsMgr was unable to set up a communications channel to %1 and there are no failover hosts. Communication will resume when %1 is available and communication from this computer is allowed.” Points to communication channel setup failure.
   • Event ID: 21006 (OpsMgr Connector): “The OpsMgr Connector could not connect to %1:%2. The error code is %3(%4). Please verify there is network connectivity, the server is running and has registered its listening port, and there are no firewalls blocking traffic to the destination.” A general connectivity error, often related to network or firewall issues.
   • Event ID: 20070 (OpsMgr Connector): “The OpsMgr Connector connected to %1, but the connection was closed immediately after authentication occurred. The most likely cause of this error is that the agent is not authorized to communicate with the server, or the server has not received configuration. Check the event log on the server for the presence of 20000 events, indicating that agents which are not approved are attempting to connect.” Suggests authentication or agent approval problems.
   • Event ID: 20051 (OpsMgr Connector): “The specified certificate could not be loaded because the certificate is not currently valid. Verify that the system time is correct and re-issue the certificate if necessary Certificate Valid Start Time : %1 Certificate Valid End Time : %2”. Indicates issues with certificate validity, commonly due to expired certificates or incorrect system time.
   • Event ID: 623 (ESE, Transaction Manager): “HealthService () The version store for instance (““) has reached its maximum size of Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back. Possible long-running transaction: SessionId: Session-context: Session-context ThreadId: . Cleanup: “. This event signals a database transaction issue within the health service’s Extensible Storage Engine (ESE) version store.

3. Guidance for Specific Events: If you encounter any of the following specific events, apply the corresponding guidelines for resolution:

   • Events 1102 and 1103: These events directly indicate that some workflows failed to load. If these are critical core system workflows, their failure can severely impact agent functionality and cause the gray state. Focus your efforts on resolving these specific workflow loading failures.
   • Events 1104, 1105, 1106, 1107, and 1108: These events are often precursors to 1102 and 1103, suggesting problems with Run As accounts. Typically, this occurs because Run As accounts are misconfigured, perhaps associated with the wrong class, or not correctly distributed to the agent. Verify the configuration and distribution settings of your Run As accounts.
   • Event 4000: This event signifies that the Monitoringhost.exe process has crashed. If the problem is due to a DLL mismatch or missing registry keys, a complete reinstallation of the agent might resolve it. If the issue persists, consider advanced troubleshooting methods:
     • Run a Process Monitor capture (using a tool like ProcMon) to log file system, registry, and process activity until the crash occurs. This can reveal the exact point of failure.
     • Generate an ADPlus dump in crash mode. This captures memory dumps of the process at the moment of failure, providing critical information for developers or advanced support teams.
   • Event ID 21006: This event clearly indicates communication problems between the agent and the management server.
     • If the agent relies on a certificate for mutual authentication, ensure the certificate has not expired and that the agent is using the correct, valid certificate.
     • If Kerberos authentication is in use, verify that the agent can successfully communicate with Active Directory for authentication purposes.
     • If authentication appears to be working, the issue might be network-related, preventing packets from reaching the management server or gateway. Try to establish a basic Telnet connection to port 5723 from the agent to the management server to test basic connectivity.
     • Additionally, run a simultaneous network trace between the agent and the management server while attempting to reproduce the communication failures. This trace can help determine if packets are reaching the management server and if any intermediate network devices (like firewalls or load balancers) are dropping or optimizing traffic.
   • Event ID 623: This event is typically observed in large Operations Manager environments where a management server or agent manages a high volume of workflows. It indicates that the Extensible Storage Engine (ESE) version store has reached its maximum size, often due to long-running transactions. This can lead to updates being rejected and impact overall health service performance. Refer to specific Operations Manager documentation for managing ESE version store issues in high-volume environments.

Scenario 3: All Agents Reporting to a Specific Management Server or Gateway Are Unavailable¶

In this scenario, all agents configured to report to a particular management server or gateway are simultaneously unavailable. This indicates that the problem likely resides with the management server or gateway itself, rather than with individual agents. Focusing your troubleshooting efforts on this central point is key to resolution.

Resolution for Scenario 3¶

To resolve a situation where all agents reporting to a specific management server or gateway are grayed out, follow these systematic troubleshooting steps:

1. Assess Workloads: Begin by determining the types of workloads that the affected management server or gateway is responsible for monitoring. This includes network devices, cross-platform agents (Unix/Linux), synthetic transactions, Windows agents, and agentless computers. An overloaded server might struggle with too many diverse monitoring tasks.
2. Verify Health Service on Parent: Confirm that the Health Service (System Center Management service) is actively running on the affected management server or gateway. If this core service is stopped, all dependent agents will lose communication.
3. Check Maintenance Mode: Determine if the management server itself has inadvertently been placed into maintenance mode. If it is, remove the server from maintenance mode, as this will prevent it from processing agent data and receiving heartbeats.
4. Examine Operations Manager Event Log (Parent Server): Review the Operations Manager event log on the management server or gateway for any of the event IDs listed in Scenario 2. Pay particular attention to Event ID 21006. If this event is present, apply the same guidelines outlined in the Resolution for Scenario 2. In this specific context, Event ID 21006 on a gateway or management server suggests that the component cannot communicate with its own parent server (which could be another management server for a gateway).

5. Look for Performance-Related Events: Examine the Operations Manager event log on the management server or gateway for the following events. These events typically signal performance issues either on the management server itself or on the Microsoft SQL Server instance hosting the OperationsManager or OperationsManagerDW databases.

   • Event ID: 2115 (HealthService): “A Bind Data Source in Management Group %1 has posted items to the workflow, but has not received a response in %5 seconds. This indicates a performance or functional problem with the workflow. Workflow Id : %2 Instance : %3 Instance Id : %4”. This suggests that a workflow is not processing data in a timely manner.
   • Event ID: 5300 (HealthService): “Local health service is not healthy. Entity state change flow is stalled with pending acknowledgement. Management Group: %2 Management Group ID: %1”. Indicates the local health service is unhealthy, and state changes are stuck.
   • Event ID: 4506 (HealthService): “Data was dropped due to too much outstanding data in rule ‘%2’ running for instance ‘%3’ with id:’%4’ in management group ‘%1’”. This event signifies that the management server is overwhelmed and dropping data due to high volume.
   • Event ID: 31551 (Health Service Modules): “Failed to store data in the Data Warehouse. The operation will be retried. Exception ‘%5’: %6. One or more workflows were affected by this. Workflow name: %2 Instance name: %3 Instance ID: %4 Management group: %1”. A transient failure to store data in the Data Warehouse.
   • Event ID: 31552 (Health Service Modules): “Failed to store data in the Data Warehouse. Exception ‘%5’: %6. One or more workflows were affected by this. Workflow name: %2 Instance name: %3 Instance ID: %4 Management group: %1”. A persistent failure to store data in the Data Warehouse.
   • Event ID: 31553 (Health Service Modules): “Data was written to the Data Warehouse staging area but processing failed on one of the subsequent operations. Exception ‘%5’: %6. One or more workflows were affected by this. Workflow name: %2 Instance name: %3 Instance ID: %4 Management group: %1”. Indicates that data reached the staging area but failed during later processing.
   • Event ID: 31557 (Health Service Modules): “Failed to obtain synchronization process state information from Data Warehouse database. The operation will be retried. Exception ‘%5’: %6. One or more workflows were affected by this. Workflow name: %2 Instance name: %3 Instance ID: %4 Management group: %1”. Failure to synchronize state with the Data Warehouse.

6. Review Run As Account Permissions: Event IDs in the 3155X range can also be triggered by incorrect Run As account configurations or insufficient permissions for these accounts on the Data Warehouse database. Verify that all required Run As accounts have the necessary database roles and permissions as specified in Operations Manager documentation.

Scenarios 4: Intermittent Environment-Wide Gray States¶

In this challenging scenario, all agents reporting to a particular management server, or even all agents across the entire Operations Manager environment, intermittently switch between healthy and gray states. This pattern suggests a systemic issue that impacts the overall health and stability of the management group. These temporary fluctuations can be difficult to diagnose without careful observation and data collection.

Resolution for Scenario 4¶

Resolving intermittent, environment-wide gray states requires first identifying the underlying cause of the temporary server unavailability. These fluctuations can stem from several common sources. Understanding the duration and timing of these events is paramount for quickly narrowing down the problem’s scope.

Common causes of temporary server unavailability include:

• Temporary Server Offline Status: The parent management server or gateway responsible for the agents may have experienced a brief outage or restart, leading to a temporary loss of communication. While often self-recovering, frequent occurrences warrant investigation into server stability.
• Agent Data Flooding: Agents might be overwhelming the management server with an excessive volume of operational data, such as alerts, state changes, or discoveries. This surge in data can lead to increased utilization of system resources on both the Operations Manager database and the management servers, causing performance bottlenecks and delays.
• Network Outages: Brief or intermittent network outages can cause temporary communication failures between the parent server and its agents. These issues might be difficult to detect without detailed network monitoring.
• Management Pack (MP) Changes: Modifications to management packs, such as new imports or updates, necessitate an Operations Manager configuration update and redistribution to agents. If these changes affect a large number of agents, the process can significantly increase resource utilization on the Operations Manager database and management servers during the redistribution phase, leading to temporary instability.

The key to effective troubleshooting in these scenarios is to thoroughly understand the exact duration of the server unavailability and the specific time of day during which it occurred. Analyzing these patterns will help you to quickly narrow the scope of the problem to specific events, schedules, or periods of high activity. Collecting performance data during these periods is crucial for identifying resource exhaustion or other contributing factors.

Troubleshooting Management Server and Gateway Performance¶

Performance issues on your management servers and gateways can directly lead to agents appearing grayed out. Understanding the typical bottlenecks for these components is crucial for effective troubleshooting. Monitoring key performance indicators helps identify when these systems are under stress.

Management Server¶

Management servers play a dual role in processing configuration updates and operational data. During a configuration update burst, which might be triggered by a management pack import or a large-scale discovery, the primary bottlenecks are typically the CPU and, secondarily, the Operations Manager installation disk I/O. The management server is responsible for compiling and forwarding updated configuration files to all targeted agents, a process that can be resource-intensive.

For operational data collection, such as incoming alerts, events, and performance data, bottlenecks are more commonly caused by the CPU. While disk I/O may also reach its maximum capacity, it is generally less likely to be the primary constraint during data ingestion. The management server’s tasks include decompressing and decrypting the incoming operational data, then efficiently inserting it into the Operational database. It also sends acknowledgments (ACKs) back to agents or gateways upon successful data receipt, utilizing disk queuing for temporary storage of these outgoing ACKs.

Gateway¶

Gateway servers are critical for expanding Operations Manager monitoring to untrusted domains and remote networks. They can become both CPU-bound and I/O-bound when relaying a substantial amount of data. Most of the CPU utilization on a gateway is attributed to the processes of decompression, compression, encryption, and decryption of incoming data, in addition to the actual transfer of that data to the management server.

All data received by the gateway from agents is stored in a persistent queue on disk before being read and forwarded to the management server by the gateway’s Health service. This persistent queuing mechanism can lead to heavy disk usage, especially when the gateway is temporarily taken offline. Upon returning online, the gateway must then process the accumulated agent data that agents attempted to send during its downtime, resulting in significant resource spikes.

To effectively troubleshoot performance issues in this situation, it is essential to collect detailed information for each affected management server or gateway. This data provides a foundational understanding of the environment and potential resource constraints.

• Exact Windows version, edition, and build number: Critical for identifying OS-level known issues or hotfix requirements.
• Number of processors: Indicates available CPU capacity.
• Amount of RAM: Essential for assessing memory availability.
• Drive that contains the Health Service State folder: Helps pinpoint potential I/O bottlenecks or configuration issues.
• Whether the antivirus software is configured to exclude the Health Service store: Antivirus interference is a common cause of performance problems and should be verified.
• RAID level (0, 1, 5, 0+1 or 1+0) for the drive used by the Health Service State: Impacts disk performance and fault tolerance.
• Number of disks used for the RAID: Contributes to the overall I/O capacity of the storage.
• Whether battery-backed write cache is enabled on the array controller: Significantly affects write performance and data integrity.

Troubleshooting SQL Server Performance¶

The underlying SQL Server databases are central to Operations Manager functionality. Performance bottlenecks on these servers can severely impact the entire management group, leading to agents appearing grayed out due to delays in data processing or configuration distribution. A thorough analysis of SQL Server performance is crucial for maintaining a healthy Operations Manager environment.

Operational Database (OperationsManager)¶

For the OperationsManager database, the most probable bottleneck is often the disk array. If the disk array is not already operating at its maximum I/O capacity, the next most likely bottleneck shifts to the CPU. The database will periodically experience slowdowns and “operational data storms,” characterized by high incidences of events, alerts, and performance data or state changes that persist for extended periods. However, a brief burst of data typically does not cause significant, prolonged delays.

During operational data insertion, the database disks are primarily engaged in write operations. CPU utilization, on the other hand, is driven by SQL Server churn, which encompasses tasks like large and complex query executions, heavy data insertion, and the grooming of large tables. By default, grooming operations often occur at midnight. While grooming large event and performance data tables usually doesn’t consume excessive CPU or disk resources, the grooming of alert and state change tables can be particularly CPU-intensive, especially for very large datasets.

The database also becomes CPU-bound during configuration redistribution bursts, which are commonly triggered by management pack imports or significant changes in the instance space. In such scenarios, the Config service actively queries the database for new agent configurations. This process typically causes noticeable CPU spikes on the database server before the service transmits these configuration updates to the agents.

Data Warehouse (OperationsManagerDW)¶

For the OperationsManagerDW database, the primary bottleneck is most frequently the disk array. This often occurs due to the insertion of large volumes of operational data. In these situations, the disks are predominantly busy performing write operations. Typically, the disks perform relatively few read operations, with exceptions being for manually generated Reporting views, which run specific queries against the data warehouse.

CPU usage in the Data Warehouse is primarily driven by SQL Server churn. CPU spikes can occur during intensive partitioning activity, especially when tables grow very large and are subsequently partitioned. Generating complex reports can also lead to increased CPU utilization. Furthermore, if there are large amounts of alerts in the operational database, the data warehouse must constantly synchronize with these, contributing to sustained CPU load.

General Troubleshooting¶

To effectively troubleshoot SQL Server performance issues, collect the following critical information for each affected SQL Server instance hosting Operations Manager databases. This data will provide a comprehensive overview of the system’s configuration and potential bottlenecks.

• Exact Windows version, edition, and build number: Provides insight into the operating system environment.
• Number of processors: Indicates the available processing power.
• Amount of RAM: Essential for assessing memory resources.
• Amount of memory allocated to SQL Server: Crucial for understanding SQL Server’s memory footprint.

• Whether SQL Server is 32-bit and AWE enabled: Relevant for older systems managing large memory footprints.

  You can find most of this information within SQL Server Management Studio or SQL Server Enterprise Manager. Access the Properties window of the server, then navigate to the General and Memory tabs. The General tab will display the SQL Server version, Windows version, platform, total RAM, and number of processors. The Memory tab shows the memory allocated to SQL Server, and in SQL Server 2008, it also includes the AWE option.

  If the operating system is 32-bit and has 4 GB or more of RAM, verify the presence of the /pae or /3gb switches in the Boot.ini file. These options might be misconfigured if the server was initially installed with 4 GB or less RAM and later upgraded. For 32-bit servers with 4 GB of RAM, the /3gb switch increases the memory SQL Server can address from 2 GB to 3 GB. However, for 32-bit servers with more than 4 GB of RAM, the /3gb switch could inadvertently limit SQL Server’s addressable memory. For these systems, add the /pae switch to Boot.ini and then enable AWE within SQL Server.

  On a multi-processor system, review the Max Degree of Parallelism (MAXDOP) setting. In SQL Server 2008, this option is located on the Advanced tab within the server’s Properties dialog box. The default value of 0 means all available processors will be utilized. A setting of 0 is generally acceptable for servers with eight or fewer processors. For servers with more than eight processors, the overhead of SQL Server coordinating all processors can become counterproductive. Therefore, for systems exceeding eight processors, it is generally recommended to set Max Degree of Parallelism to a value of 8. To apply this change, execute the following commands in SQL Query Analyzer:

  sp_configure 'show advanced options', 1
  GO
  RECONFIGURE WITH OVERRIDE
  GO
  sp_configure 'max degree of parallelism', 8
  GO
  RECONFIGURE WITH OVERRIDE
  GO
  

• Drive letters that contain data warehouse, Operations Manager DB, and Tempdb files: Helps map databases to physical storage.

• Whether the antivirus software is configured to exclude SQL data and log files: Antivirus scanning of SQL database files can severely degrade performance.
• Amount of free space on drives that contain data warehouse, Operations Manager DB, and Tempdb files: Insufficient free space can lead to performance issues and database errors.
• Storage type (SAN or local): Influences I/O characteristics and potential troubleshooting paths.
• RAID level (0, 1, 5, 0+1 or 1+0) for drives that are used by SQL Server: Determines the performance and redundancy of the storage.
• If SAN storage is used: number of spindles on each LUN that’s used by SQL Server: Critical for calculating theoretical I/O capacity.

• If the converted Exchange 2007 management pack is being used or has ever been used: number of rows in the LocalizedText table in the Operations Manager database and in the EventPublisher table in the data warehouse database: High row counts in these tables can indicate specific performance issues related to this MP.

  To determine the row amounts, execute the following commands in SQL Query Analyzer:

  USE OperationsManager SELECT COUNT(*) FROM LocalizedText
  USE OperationsManagerDW SELECT COUNT(*) FROM EventPublisher
  

Counters to Identify Memory Pressure¶

Monitoring specific performance counters can help identify if your SQL Server is experiencing memory pressure, which can directly impact Operations Manager performance. These counters provide insights into how efficiently SQL Server is using available memory.

Performance Counter Name	Description
MSSQL$<instance>: Buffer Manager: Page life expectancy	This counter measures how long data pages persist in the buffer pool, in seconds. If this value consistently falls below 300 seconds, it may strongly indicate that the server could benefit from more physical memory. It can also be a symptom of index fragmentation, which forces more frequent page reads from disk.
MSSQL$<instance>: Buffer Manager: Lazy writes/sec	The lazy writer process is responsible for freeing up space in the buffer pool by moving modified data pages (dirty pages) to disk. Generally, the value for this counter should not consistently exceed 20 writes per second. Ideally, for optimal performance, this value should be close to zero.
Memory: Available Mbytes	This counter reports the amount of physical memory, in megabytes, that is immediately available for allocation to processes or for system use. Values consistently below 100 MB can indicate memory pressure on the operating system. If this amount drops below 10 MB, memory pressure is clearly and significantly present.
Process: Private Bytes: _Total	This counter represents the total amount of memory (including both physical RAM and virtual memory from the page file) currently being used by all running processes combined.
Process: Working Set: _Total	This counter indicates the total amount of physical memory (RAM) actively being used by all running processes combined. If the value for Process: Working Set: _Total is significantly lower than Process: Private Bytes: _Total, it suggests that processes are actively paging data to and from disk too heavily. A difference exceeding 10% is generally considered significant.

Counters to Identify Disk Pressure¶

Disk I/O is a frequent bottleneck for database servers. By capturing and analyzing these physical disk counters for all drives hosting SQL data or log files, you can pinpoint disk pressure and potential performance limitations.

• % Idle Time: This counter measures the percentage of time the disk was idle during the sample interval. Any consistent reading below 50 percent can indicate a disk bottleneck, as the disk is spending more than half its time actively processing requests.
• Avg. Disk Queue Length: This value represents the average number of read and write requests that were queued for the selected disk during the sample interval. Ideally, this value should not exceed twice the number of physical spindles on a LUN. For instance, if a LUN consists of 25 spindles, an Avg. Disk Queue Length of 50 is acceptable. However, if a LUN has only 10 spindles, a value of 25 would be considered too high, indicating a severe bottleneck. You can use the following formulas based on your RAID level and number of disks:
  • RAID 0: All disks in a RAID 0 set actively participate in I/O operations.
    Average Disk Queue Length <= # (Disks in the array) * 2
  • RAID 1: Approximately half of the disks perform active work; therefore, only half of them should be counted towards the disk queue length calculation.
    Average Disk Queue Length <= # (Disks in the array / 2) * 2
  • RAID 10: Similar to RAID 1, half of the disks are involved in active I/O.
    Average Disk Queue Length <= # (Disks in the array / 2) * 2
  • RAID 5: All disks in a RAID 5 set contribute to I/O operations.
    Average Disk Queue Length <= # Disks in the array * 2
• Avg. Disk sec/Transfer: This counter measures the average time, in seconds, it takes to complete one disk I/O operation (read or write).
• Avg. Disk sec/Read: This counter specifically measures the average time, in seconds, to read data from the disk.

• Avg. Disk sec/Write: This counter specifically measures the average time, in seconds, to write data to the disk.

  The last three counters in this list (Avg. Disk sec/Transfer, Avg. Disk sec/Read, Avg. Disk sec/Write) should consistently maintain values of approximately .020 seconds (20 milliseconds) or lower. They should ideally never exceed .050 seconds (50 milliseconds). According to established SQL Server performance troubleshooting guidelines, the following thresholds are generally accepted:
  * Less than 10 ms: Very good performance.
  * Between 10 - 20 ms: Acceptable performance.
  * Between 20 - 50 ms: Slow performance, requires attention and investigation.
  * Greater than 50 ms: Indicates a serious I/O bottleneck that needs immediate resolution.

• Disk Bytes/sec: This counter shows the number of bytes being transferred to or from the disk per second. It provides a measure of the disk’s throughput in terms of data volume.

• Disk Transfers/sec: This counter indicates the number of input and output operations per second (IOPS) performed by the disk. It is a critical metric for understanding the disk’s capability to handle concurrent requests.

When % Idle Time is consistently low (10 percent or less), it signifies that the disk is fully utilized and acting as a bottleneck. In such cases, Disk Bytes/sec and Disk Transfers/sec provide excellent indicators of the drive’s maximum achievable throughput in terms of data volume and IOPS, respectively. The actual throughput of a SAN drive can vary significantly based on factors such as the number of spindles, the speed of the drives, and the channel speed. It is best practice to consult with your SAN vendor to ascertain the expected maximum bytes and IOPS your specific drive configuration should support. If the % Idle Time is low, but the observed values for Disk Bytes/sec and Disk Transfers/sec do not meet the drive’s expected throughput, engaging your SAN vendor for further troubleshooting is highly recommended.

Operations Manager Performance Counters¶

Beyond SQL Server, specific performance counters within Operations Manager itself offer granular insights into the health and efficiency of your management and gateway servers. Monitoring these counters can help identify bottlenecks directly related to the Operations Manager components.

Gateway Server Role¶

Gateway servers are crucial for extending Operations Manager monitoring. Monitoring their performance ensures they can efficiently handle agent communication.

Overall Performance Counters¶

These counters provide a high-level overview of the gateway server’s resource utilization.

Performance Counter Name
Processor(_Total)\% Processor Time
Memory\% Committed Bytes In Use
Network Interface(*)\Bytes Total/sec
LogicalDisk(*)\% Idle Time
LogicalDisk(*)\Avg. Disk Queue Length

Operations Manager Process Generic Performance Counters¶

These counters focus on the resource consumption of the core Operations Manager processes running on the gateway.

Performance Counter Name	Description
Process(HealthService)\% Processor Time	Measures the CPU utilization by the Health Service process.
Process(HealthService)\Private Bytes	Indicates the amount of private memory (non-shared) consumed by the Health Service. This value can vary significantly, often several hundred megabytes, depending on the number of agents managed by the gateway.
Process(HealthService)\Thread Count	Shows the number of threads currently active within the Health Service process.
Process(HealthService)\Virtual Bytes	Represents the total virtual memory space reserved by the Health Service process.
Process(HealthService)\Working Set	Displays the amount of physical memory (RAM) currently in use by the Health Service process.
Process(MonitoringHost*)\% Processor Time	Measures the CPU utilization of all MonitoringHost processes.
Process(MonitoringHost*)\Private Bytes	Indicates the private memory consumed by MonitoringHost processes.
Process(MonitoringHost*)\Thread Count	Shows the number of threads within MonitoringHost processes.
Process(MonitoringHost*)\Virtual Bytes	Represents the virtual memory space reserved by MonitoringHost processes.
Process(MonitoringHost*)\Working Set	Displays the physical memory (RAM) used by MonitoringHost processes.

Operations Manager Specific Performance Counters¶

These counters are directly related to the internal workings and communication efficiency of Operations Manager components on the gateway.

Performance Counter Name	Description
Health Service\Workflow Count	Represents the total number of workflows currently active and running within the Health Service.
Health Service Management Groups(*)\Active File Uploads	This counter indicates the number of ongoing file transfers that the gateway is actively handling. It typically represents the number of management pack files being uploaded to agents. If this value remains consistently high for an extended period, especially without significant management pack imports occurring, it may signal an issue affecting file transfer operations and potentially cause delays in configuration distribution.
Health Service Management Groups(*)\Send Queue % Used	This counter measures the percentage of the persistent queue space that is currently in use. If this value consistently stays above 10% for a long duration and does not decrease, it strongly indicates that the queue is backed up. This condition is often a result of an overloaded Operations Manager system, implying that the management server or database is either too busy to process incoming data efficiently or is currently offline.
OpsMgr Connector\Bytes Received	Shows the number of network bytes received by the gateway. This represents the amount of incoming network traffic before any decompression occurs.
OpsMgr Connector\Bytes Transmitted	Displays the number of network bytes sent by the gateway. This represents the amount of outgoing network traffic after any compression has been applied.
OpsMgr Connector\Data Bytes Received	Indicates the number of data bytes received by the gateway. This represents the amount of incoming data after it has been decompressed.
OpsMgr Connector\Data Bytes Transmitted	Shows the number of data bytes sent by the gateway. This represents the amount of outgoing data before compression is applied.
OpsMgr Connector\Open Connections	Represents the total number of open connections currently active on the gateway. This number should ideally match the total count of agents or management servers that are directly connected to this specific gateway.

Management Server Role¶

Management servers are the heart of your Operations Manager environment. Monitoring their performance is crucial for the overall health of your management group.

Overall Performance Counters¶

These counters provide a general overview of the management server’s resource health.

Performance Counter Name
Processor(_Total)\% Processor Time
Memory\% Committed Bytes In Use
Network Interface(*)\Bytes Total/sec
LogicalDisk(*)\% Idle Time
LogicalDisk(*)\Avg. Disk Queue Length

Operations Manager Process Generic Performance Counters¶

These counters specifically track the resource consumption of the core Operations Manager processes running on the management server.

Performance Counter Name	Description
Process(HealthService)\% Processor Time	Measures the CPU utilization by the Health Service process.
Process(HealthService)\Private Bytes	Indicates the amount of private memory consumed by the Health Service. This value can vary significantly, often several hundred megabytes, depending on the number of agents and workflows managed by this server.
Process(HealthService)\Thread Count	Shows the number of threads currently active within the Health Service process.
Process(HealthService)\Virtual Bytes	Represents the total virtual memory space reserved by the Health Service process.
Process(HealthService)\Working Set	Displays the amount of physical memory (RAM) actively in use by the Health Service process.
Process(MonitoringHost*)\% Processor Time	Measures the CPU utilization of all MonitoringHost processes.
Process(MonitoringHost*)\Private Bytes	Indicates the private memory consumed by MonitoringHost processes.
Process(MonitoringHost*)\Thread Count	Shows the number of threads within MonitoringHost processes.
Process(MonitoringHost*)\Virtual Bytes	Represents the virtual memory space reserved by MonitoringHost processes.
Process(MonitoringHost*)\Working Set	Displays the physical memory (RAM) used by MonitoringHost processes.

Operations Manager Specific Performance Counters¶

These counters provide granular details on the performance of specific Operations Manager components and communication aspects on the management server.

| Performance Counter Name | Description | Performance Counter Name | Description |
| Health Service\\Workflow Count | This counter represents the total number of active workflows currently being processed by the Health Service. |
| Health Service Management Groups(*)\Active File Uploads| This counter measures the current number of ongoing file transfers handled by the gateway, primarily representing management pack files being uploaded to agents. A consistently high value over an extended period, particularly without active management pack imports, may indicate a file transfer problem. It can also signify a backlog or slowdown in the distribution of configuration updates. |
| Health Service Management Groups(*)\Send Queue % Used| This counter indicates the percentage of the persistent queue that is currently in use by the Health Service. A sustained value above 10% that doesn’t decrease suggests a backlog in the queue. This condition often arises when the Operations Manager system is overloaded, or the management server or database is too busy or temporarily offline. This prevents data from being processed and forwarded in a timely manner. |
| OpsMgr Connector\Bytes Received | Displays the number of network bytes received by the management server, representing incoming network traffic before any decompression. |
| Health Service Management Groups(*)\Send Queue % Used| This counter measures the current percentage of the persistent queue utilized by the Health Service. If this value consistently stays above 10% for an extended period without decreasing, it indicates a queue backlog. This typically points to an overloaded Operations Manager system where the management server or database is too busy, or temporarily offline, preventing efficient data processing and forwarding. |
| OpsMgr Connector\Bytes Received | Displays the number of network bytes received by the management server, representing incoming network traffic before any decompression occurs. |
| Health Service\Workflow Count | This counter measures the total number of active workflows currently being managed by the Health Service on the management server. |
| Health Service Management Groups(*)\Send Queue % Used| This counter measures the percentage of the persistent queue currently in use by the Health Service. A sustained value above 10% that does not decrease indicates a queue backlog. This typically signifies an overloaded Operations Manager system, where the management server or database is either too busy to process incoming data efficiently or is offline, preventing timely data forwarding. |
| OpsMgr Connector\Bytes Received | Displays the number of network bytes received by the management server, representing the incoming network traffic before any decompression. |
| Health Service Management Groups(*)\Send Queue % Used| This counter measures the percentage of the persistent queue currently in use by the Health Service. A sustained value above 10% that does not decrease indicates a queue backlog. This typically signifies an overloaded Operations Manager system, where the management server or database is either too busy to process incoming data efficiently or is offline, preventing timely data forwarding. |
| OpsMgr Connector\Bytes Received | Displays the number of network bytes received by the management server, representing incoming network traffic before any decompression. |
| OpsMgr Connector\Bytes Transmitted | The number of network bytes sent by the management server, representing the outgoing network traffic after any compression. |
| OpsMgr Connector\Data Bytes Transmitted | Represents the total amount of data bytes sent by the management server, before any compression is applied. |
| OpsMgr Connector\Open Connections | Represents the number of open connections currently established on the management server. This count should ideally correspond to the number of agents and gateway servers that are directly connected to this specific management server. |

---

We hope this comprehensive guide has provided valuable insights into troubleshooting grayed-out agents, management servers, and gateways in System Center Operations Manager. Effective monitoring and a proactive approach to performance management are critical for a healthy OpsMgr environment.

What are your experiences with grayed-out agents in Operations Manager? Have you encountered specific scenarios not covered here, or do you have any unique troubleshooting tips you’d like to share? Please feel free to leave a comment below and join the conversation! Your insights can help other administrators facing similar challenges.