Proactive Configuration Change Management: Detect and Troubleshoot Issues in Operations Manager

Table of Contents

This article delves into the methods for detecting and resolving frequent configuration changes within System Center Operations Manager. Understanding and managing configuration changes is crucial for maintaining a stable and efficiently operating monitoring environment. Excessive configuration modifications, often termed “configuration churn,” can lead to instability and performance issues. This guide provides insights into identifying the causes of configuration churn and offers strategies for effective troubleshooting and mitigation.

Configuration Overview¶

The System Center Management Configuration service plays a pivotal role in Operations Manager. It is the core component responsible for calculating the configuration for every health service within the management group. This configuration encompasses a comprehensive set of elements, including rules, monitors, discoveries, and tasks. These elements are not only for the health service itself but also for all the instances that the health service is tasked with monitoring. Essentially, the Configuration service orchestrates how each agent and management server behaves and what it monitors.

To accurately compute all necessary configurations for each health service, the Management Configuration service requires a detailed inventory of several key components. These include:

• All instances of all monitored classes: This is the complete list of all objects being monitored, categorized by their class (e.g., Windows Server, SQL Database).
• Hosting relationships between instances: This defines how monitored objects are related to each other, such as a database hosted on a server. Understanding these relationships is vital for proper monitoring.
• Rules, monitors, discoveries, and other workflows assigned to monitored classes: These are the monitoring logic and actions that are applied to each class of monitored objects. They dictate what is collected, how health is assessed, and what actions are taken.
• Health services responsible for monitoring the instances: This identifies which agents or management servers are responsible for monitoring specific instances, ensuring proper delegation of monitoring tasks.

Furthermore, the Management Configuration service needs to access the membership of all instance groups within the management group. Instance groups are dynamic collections of objects, and their membership can change. The service must also apply any overrides that are defined for rules and monitors targeting these groups, classes, or individual instances. Overrides allow for customization of monitoring behavior for specific scopes.

Objects within a management group are classified as instances of monitored classes based on discovery data submitted by discovery workflows. Discovery workflows are responsible for identifying and defining the objects to be monitored. A significant aspect of this process is that if a key property of an object changes, it can be treated in different ways. It might be added as a new instance of a monitored class if the change is substantial enough. Alternatively, if certain properties change, the object might no longer be considered an instance of that class, effectively being removed from monitoring.

As the class membership of an object evolves, the configuration for the health service monitoring that object also changes dynamically. These configuration shifts manifest as rules, monitors, discoveries, tasks, and overrides being added to or removed from the health service’s configuration. This constant adaptation is necessary to ensure that monitoring accurately reflects the current state of the IT environment.

Configuration Churn¶

Agents in Operations Manager rely on receiving a stable configuration to function correctly. However, in certain situations, agents may struggle to obtain or maintain a stable configuration. This instability is often referred to as “configuration churn”. Several scenarios can lead to this undesirable state:

• Large Volume of Discovery Data Submission: When a substantial amount of discovery data is simultaneously submitted to the Management Configuration service, it can become overwhelmed. Processing this massive influx of data requires significant resources and time.

• Rapid Discovery Data Submission: If discovery data is submitted at a rate that exceeds the Management Configuration service’s processing capacity, a backlog occurs. The service is perpetually in a state of processing data, never quite catching up before more data arrives. This constant influx keeps the configuration in a state of flux.

This frequent submission of discovery data, or configuration churn, has detrimental effects on the Operations Manager environment. Agents might operate under outdated configurations, leading to inaccurate monitoring and alerting. Management servers themselves can also suffer from stale configurations, impacting their performance and reliability. A visible symptom of configuration churn in the Operations console is that some health services may appear dimmed or unavailable, signaling a problem.

Discovery data originates from health services executing discovery workflows. The introduction of a new management pack into a management group can trigger numerous discovery workflows to run across all agents. Management packs define how different applications and services are monitored. Furthermore, as new instances of monitored objects are discovered, additional discovery workflows may be initiated on specific agents to gather detailed information about these new objects. Changes to groups, overrides, and other workflow configurations can also trigger discovery workflows to rerun on agents to reflect the updated monitoring policies. Adding new agents to the environment also necessitates the Management Configuration service to update its understanding of the instance space based on the new agent’s configuration and the objects it discovers.

The Configuration Management service is forced to recalculate health service configurations frequently under specific conditions. These conditions often point to misconfigurations or inefficient monitoring practices:

• Overly Frequent Discovery Workflow Execution: If a discovery workflow is configured to run too frequently, for example, every few minutes, it generates a constant stream of discovery data, even if the underlying environment is relatively stable.

• Volatile Properties in Discovery Workflows: If a discovery workflow is designed to discover properties that change very frequently (volatile properties), every execution of the workflow will likely result in changes being reported. Properties like free disk space are examples of volatile properties.

When these scenarios are widespread across many agents, or if management servers are already under heavy load due to other monitoring activities, the Configuration Management service can become unable to keep pace with the rate of change. This inability to keep up leads to configuration churn, creating a cycle of instability and impacting the overall health of the Operations Manager environment.

Identify Configuration Churn by Using the Management Server Event Log¶

The Operations Manager event log on management servers provides valuable clues for identifying configuration churn. Specific event IDs signal configuration changes and potential churn issues.

An event resembling the following indicates that the management group configuration has changed due to new discovery data:

Log Name:     Operations Manager
Source:       OpsMgr Connector
Event ID:     21024
Level:        Information
Computer:     <Name>
Description:
OpsMgr's configuration may be out-of-date for management group <ManagementGroupName>, and has requested updated configuration from the Configuration Service. The current(out-of-date) state cookie is "3A B0 1E 5C 81 F3 12 F5 56 B7 8A EF F8 01 BA 09 86 55 06 48 "

This Event ID 21024 is an informational event indicating that a management server has detected that its configuration might be outdated. It is requesting an updated configuration from the Configuration Service. The “state cookie” is an internal identifier used to track configuration versions.

Subsequently, an event similar to this indicates that the Management Configuration service has completed processing the new discovery data and calculated any necessary changes to the management group configuration:

Log Name:     Operations Manager
Source:       OpsMgr Connector
Event ID:     21025
Level:        Information
Computer:     <Name>
Description:
OpsMgr has received new configuration for management group <ManagementGroupName> from the Configuration Service. The new state cookie is "34 FA 11 61 4D B8 03 59 3D 1D 66 B7 83 F3 C0 AA 7A 6F 1A 3B "

Event ID 21025 signifies that the management server has successfully received a new configuration from the Configuration Service. The new “state cookie” reflects the updated configuration version.

In a healthy Operations Manager environment, every Event ID 21024 should ideally be followed by either Event ID 21025 or Event ID 21026. Event ID 21026 (not shown in the original text, but important to note) is similar to 21025, but it indicates that while a configuration update was requested (21024), the discovery data did not result in any actual configuration changes. In larger management groups, it’s normal to see pairs of 21024 and 21025/21026 events occurring several times per hour, reflecting routine configuration updates.

However, long sequences of Event ID 21024 events without corresponding 21025 or 21026 events are a strong indicator of configuration churn. This pattern suggests that the management server is continuously requesting configuration updates but is not receiving stable, processed configurations. The event log may also contain the following warning event, which explicitly signals detected churn:

Log Name:     Operations Manager
Source:       OpsMgr Config Service
Event ID:     29202
Level:        Warning
Computer:     <Name>
Description:
OpsMgr Config Service could not retrieve a consistent state from the OpsMgr database due to too frequent database changes.
This could be due to a normal and temporary increase of discovery data; however check the most recent changes to determine if this increase is unexpected.
Most recent monitoring object change:
Instance = %1
Class = %2
Modified time = %3
Most recent monitoring relationship change:
Relationship instance = %4
Source instance = %5
Target instance = %6
RelationshipClass = %7
Modified time = %8

Event ID 29202 is a warning from the Configuration Service itself. It directly indicates that the service is struggling to retrieve a consistent state from the Operations Manager database due to excessively frequent database modifications. This is a clear sign of configuration churn impacting the core configuration processes. The event description provides details about the most recent monitoring object and relationship changes that contributed to the churn, if available. These details can help pinpoint the source of the frequent changes. If no specific entity or relationship was being processed during the inconsistent read, these fields will be empty.

The Data Access Layer, when querying for configuration changes, needs to read data from multiple database tables. If one of these tables is modified after it has been read but before all necessary tables are read, a data inconsistency occurs. This inconsistency triggers Event ID 29202 and prompts the Data Access Layer to retry the operation.

Identify Potential Causes of Configuration Churn by Using the Operations Manager Data Warehouse¶

For management groups that have the Operations Manager Reporting component installed, the Data Warehouse provides a rich source of data for identifying the workflows responsible for frequent configuration changes. Running specific SQL queries against the Data Warehouse instance using SQL Server Management Studio can pinpoint problematic discoveries.

Query 1: Total changes submitted by discovery workflows in the last 24 hours

This query helps identify discovery workflows that are submitting the highest number of changes in a 24-hour period. This is a good starting point to find the biggest contributors to configuration churn.

select
   ManagedEntityTypeSystemName,
   DiscoverySystemName,
   count(*) As 'Changes'
from
   (
      select distinct
         MP.ManagementPackSystemName,
         MET.ManagedEntityTypeSystemName,
         PropertySystemName,
         D.DiscoverySystemName,
         D.DiscoveryDefaultName,
         MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName',
         MET1.ManagedEntityTypeDefaultName As 'TargetTypeDefaultName',
         ME.Path,
         ME.Name,
         C.OldValue,
         C.NewValue,
         C.ChangeDateTime
      from
         dbo.vManagedEntityPropertyChange C
         inner join
            dbo.vManagedEntity ME
            on ME.ManagedEntityRowId = C.ManagedEntityRowId
         inner join
            dbo.vManagedEntityTypeProperty METP
            on METP.PropertyGuid = C.PropertyGuid
         inner join
            dbo.vManagedEntityType MET
            on MET.ManagedEntityTypeRowId = ME.ManagedEntityTypeRowId
         inner join
            dbo.vManagementPack MP
            on MP.ManagementPackRowId = MET.ManagementPackRowId
         inner join
            dbo.vManagementPackVersion MPV
            on MPV.ManagementPackRowId = MP.ManagementPackRowId
         left join
            dbo.vDiscoveryManagementPackVersion DMP
            on DMP.ManagementPackVersionRowId = MPV.ManagementPackVersionRowId
            AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%' + MET.ManagedEntityTypeSystemName + '%'\
         left join
            dbo.vManagedEntityType MET1
            on MET1.ManagedEntityTypeRowId = DMP.TargetManagedEntityTypeRowId
         left join
            dbo.vDiscovery D
            on D.DiscoveryRowId = DMP.DiscoveryRowId
      where
         ChangeDateTime > dateadd(hh, - 24, getutcdate())
   )
   As # T
group by
   ManagedEntityTypeSystemName,
   DiscoverySystemName
order by
   count(*) DESC

This query produces a result set with three columns:

1. ManagedEntityTypeSystemName: The class of object targeted by the discovery workflow.
2. DiscoverySystemName: The internal name of the discovery workflow.
3. Changes: The total count of property changes for all instances of the class submitted by the workflow in the last 24 hours.

The “Changes” column is particularly important. It represents the number of times the Configuration Management service had to recalculate the configuration for an agent health service due to changes from that specific discovery workflow. A high number of changes for a particular workflow indicates it’s a significant contributor to configuration churn.

It’s important to note that in a stable environment, some level of change is normal. Activities like adding or removing properties, agents being added or decommissioned, or changes in server roles will all contribute to configuration changes. However, in environments experiencing configuration churn, one or a few workflows will likely exhibit significantly higher “Changes” values compared to others, clearly standing out as problematic.

Query 2: Properties changed in the last 24 hours

This query provides a more detailed view, showing which properties have changed, along with context information. It complements the first query by providing granularity.

select distinct
   MP.ManagementPackSystemName,
   MET.ManagedEntityTypeSystemName,
   PropertySystemName,
   D.DiscoverySystemName,
   D.DiscoveryDefaultName,
   MET1.ManagedEntityTypeSystemName As 'TargetTypeSystemName',
   MET1.ManagedEntityTypeDefaultName As 'TargetTypeDefaultName',
   ME.Path,
   ME.Name,
   C.OldValue,
   C.NewValue,
   C.ChangeDateTime
from
   dbo.vManagedEntityPropertyChange C
   inner join
      dbo.vManagedEntity ME
      on ME.ManagedEntityRowId = C.ManagedEntityRowId
   inner join
      dbo.vManagedEntityTypeProperty METP
      on METP.PropertyGuid = C.PropertyGuid
   inner join
      dbo.vManagedEntityType MET
      on MET.ManagedEntityTypeRowId = ME.ManagedEntityTypeRowId
   inner join
      dbo.vManagementPack MP
      on MP.ManagementPackRowId = MET.ManagementPackRowId
   inner join
      dbo.vManagementPackVersion MPV
      on MPV.ManagementPackRowId = MP.ManagementPackRowId
   left join
      dbo.vDiscoveryManagementPackVersion DMP
      on DMP.ManagementPackVersionRowId = MPV.ManagementPackVersionRowId
      AND CAST(DefinitionXml.query('data(/Discovery/DiscoveryTypes/DiscoveryClass/@TypeID)') AS nvarchar(max)) like '%' + MET.ManagedEntityTypeSystemName + '%'\
   left join
      dbo.vManagedEntityType MET1
      on MET1.ManagedEntityTypeRowId = DMP.TargetManagedEntityTypeRowId
   left join
      dbo.vDiscovery D
      on D.DiscoveryRowId = DMP.DiscoveryRowId
where
   ChangeDateTime > dateadd(hh, - 24, getutcdate())
ORDER BY
   MP.ManagementPackSystemName,
   MET.ManagedEntityTypeSystemName

This query output includes:

• ManagementPackSystemName: The management pack containing the discovery.
• ManagedEntityTypeSystemName: The class of object.
• PropertySystemName: The name of the property that changed.
• DiscoverySystemName: The internal name of the discovery workflow.
• DiscoveryDefaultName: The friendly name of the discovery workflow.
• TargetTypeSystemName & TargetTypeDefaultName: Target type information.
• ME.Path & ME.Name: Path and name of the managed entity (agent).
• OldValue & NewValue: The old and new values of the changed property.
• ChangeDateTime: Timestamp of the change.

By combining the insights from both queries, you can effectively pinpoint not just which discovery workflows are causing the most churn (Query 1), but also what properties are changing, on which agents, and what the old and new values are (Query 2). This detailed information is essential for targeted troubleshooting and remediation.

Reduce Configuration Churn¶

Addressing configuration churn requires identifying the root causes, often stemming from overly aggressive or poorly designed discovery workflows. Older management packs sometimes introduced discovery workflows that submitted property changes too frequently, leading to churn. Modern management packs generally incorporate improvements to mitigate this, either by reducing the frequency of data submission or by avoiding the discovery of volatile properties. Upgrading to the latest versions of management packs is a crucial first step in reducing churn.

If upgrading management packs is not immediately feasible, or if a newer version is unavailable, several mitigation strategies can be employed:

• Adjust Discovery Interval via Overrides: The execution frequency of discovery workflows can be adjusted using overrides. By increasing the interval between discovery runs, you can significantly reduce the volume of discovery data submitted and alleviate pressure on the Configuration Management service. For example, if a discovery is running every 15 minutes, consider increasing it to every hour or even longer, depending on the volatility of the monitored properties.

• Disable Problematic Discoveries via Overrides: In situations where a specific discovery workflow is identified as the primary source of configuration churn and its data is not critical for immediate monitoring, disabling the discovery entirely via overrides can be a temporary but effective solution. If a discovery is disabled for an extended period, the objects discovered by it may eventually be “groomed” (removed) from the Operations Manager database as part of data retention policies. However, disabling the discovery can provide immediate relief from churn while a permanent solution is developed.

• Periodic Rediscovery: If a discovery is disabled temporarily, it’s important to re-enable it periodically for short intervals. This allows the discovery to rediscover objects before they are groomed from the database, maintaining a baseline level of monitoring data.

• Rewrite Custom Discoveries: If the churn originates from a custom discovery workflow, particularly one targeting volatile properties like free disk space, the discovery logic should be rewritten. Discovery workflows should ideally focus on discovering relatively static properties that define the identity and type of an object, rather than properties that change frequently. Volatile data, which is not directly used in configuration calculations, is better suited for collection through performance rules rather than discovery workflows. Discovery workflows should also avoid targeting instances with very short lifecycles (weeks or less), as this can also contribute to churn.

It’s crucial to analyze the results of the SQL queries against the Data Warehouse to pinpoint the specific discovery workflows contributing to churn. Once identified, apply the appropriate mitigation strategies, starting with the least disruptive options like adjusting discovery intervals before considering disabling discoveries entirely.

Additional Performance Tuning¶

In large Operations Manager management groups, typically those with more than 1,000 agents, the root management server (RMS) can become heavily loaded. Operations that are routine in smaller environments can become bottlenecks in larger deployments. Even a moderate rate of property changes can trigger configuration churn simply because the RMS takes longer to process these changes due to its overall workload.

In such scenarios, optimizing the performance of the RMS and the management group infrastructure becomes critical. Several configuration adjustments can reduce the operational overhead on the RMS, enabling it to process property changes more efficiently and prevent configuration churn. Key performance tuning areas include:

• Database Optimizations: Ensure the Operations Manager databases (OperationsManager and OperationsManagerDW) are properly sized, indexed, and maintained. Regular database maintenance tasks like index defragmentation and statistics updates are essential. Consider using SSD storage for databases to improve I/O performance.
• Management Server Resource Allocation: Allocate sufficient CPU, memory, and disk resources to management servers, especially the RMS. Monitor server performance metrics to identify resource bottlenecks.
• Agent and Management Server Communication Tuning: Review and optimize communication settings between agents and management servers, such as heartbeat intervals and data transmission batching.
• Workflow Optimization: In addition to discovery workflows, optimize other types of workflows (rules, monitors) to reduce unnecessary data collection and processing. Review thresholds and frequencies to ensure they are appropriate for the environment.
• Management Pack Pruning: Regularly review and remove management packs that are no longer needed or relevant. Unnecessary management packs contribute to configuration complexity and processing overhead.

Detailed performance optimization guidance for Operations Manager 2007 R2 and 2012 (which are still relevant for later versions in terms of core concepts) can be found in resources like the Microsoft documentation and community articles. Implementing these performance optimizations can significantly improve the RMS’s ability to handle configuration changes and reduce the likelihood of churn in large environments.

Force Configuration Change for the Management Group¶

In persistent configuration churn situations, where the churn is continuous and prevents any mitigation efforts (like disabling problematic workflows or reducing their frequency) from being effectively propagated to agents, a more drastic measure may be necessary. In these cases, it might be required to temporarily block the flow of incoming discovery data to allow the System Center Configuration Management service to calculate a stable, current configuration. This stable configuration should incorporate the changes intended to reduce churn, such as disabled or less frequent workflows.

Discovery data from agents and other management servers is submitted to the OperationsManager database via the System Center Data Access Service (DAS). The initial submission point is the DAS on the RMS, which receives data from agents and other management servers. To temporarily halt this data flow, you can use the Windows Firewall or other network security mechanisms to block incoming connections to the RMS specifically on port 5723. Port 5723 is the default port used by Operations Manager agents and management servers to communicate with the RMS.

Blocking port 5723 on the RMS effectively prevents new discovery data from reaching the OperationsManager database. This pause gives the Configuration Management service the opportunity to catch up, process existing data, and calculate a current configuration that reflects the desired changes (e.g., disabled workflows).

Crucially, while blocking port 5723, do not stop or disable the System Center Management service or the System Center Data Access Service on the RMS. These services must remain running for the Configuration Management service to successfully calculate the management group configuration. The Configuration Management service depends on:

• A running and healthy System Center Management service on the RMS.
• Functional communication between the System Center Data Access Service and the Operations Manager database.

Once port 5723 is blocked, monitor the Operations Manager event log on the RMS. As soon as you see Event ID 21025 (indicating successful configuration update), immediately lift the firewall block or port exclusion on port 5723. Event ID 21025 signals that the Configuration Management service has calculated a new, stable configuration for the management group, which now includes the intended changes to address churn.

Note that while port 5723 is blocked, data may accumulate (backlog) on agents and other management servers as they are unable to submit it to the RMS. Therefore, it’s essential to remove the block as soon as Event ID 21025 is observed to allow this backlogged data to flow into the system and resume normal operations. This forced configuration change procedure is a temporary intervention to break the churn cycle and allow mitigation strategies to take effect.

Identify Potential Causes of Configuration Churn by Using Operations Manager Reporting¶

Operations Manager reporting offers valuable tools for understanding data volumes and identifying potential sources of configuration churn over time. New reports have been introduced specifically to provide insights into the overall data processing volume within a management group. These reports can be used to establish a baseline for normal data volumes and to identify workflows that are generating excessive data, potentially contributing to churn. Once configuration churn issues are addressed, these reports become essential for long-term monitoring and proactive planning to prevent future recurrences of churn.

Two key reports are particularly relevant for analyzing data volumes and churn:

• Data Volume by Management Pack Report:

  This report provides a summarized view of data volume, broken down by management pack. It lists the number of occurrences for various data types generated by each management pack:

  • Discoveries: Number of discovery data submissions.
  • Alerts: Number of alerts generated.
  • Performance: Number of performance data samples submitted (instances for performance counters collected by the management pack).
  • Events: Number of events collected.
  • State Changes: Number of state changes reported.

  By examining this report, you can quickly identify management packs that are generating disproportionately high volumes of data across different categories. Management packs with high discovery counts might be candidates for further investigation regarding their discovery workflow design and frequency.

• Data Volume by Workflow and Instance Report:

  This report provides a more granular view, detailing data volume by individual workflows (discoveries, rules, monitors, etc.) and instances of monitored objects. It allows you to drill down into specific workflows and instances to understand their data contribution.

  There are two ways to access this report:

  1. Drill-down from “Data Volume by Management Pack” report: In the “Data Volume by Management Pack” report, you can select any of the count cells in the table (e.g., the “Discoveries” count for a specific management pack). This action will directly open the “Data Volume by Workflow and Instance” report, pre-filtered to show data for the selected management pack. This is a convenient way to investigate high-volume management packs identified in the summary report.

  2. Run directly from the Reporting section: You can also run the “Data Volume by Workflow and Instance” report directly from the “Reporting” section in the Operations console. However, when running it directly, it’s crucial to configure the report parameters appropriately to customize the results. The default parameters might not provide the specific information you are looking for. This report is designed to provide detailed context to the “Data Volume by Management Pack” report, so understanding the summary data first is often helpful in setting the parameters effectively.

Using these reporting tools regularly allows you to establish a baseline of normal data volumes for your management group. Deviations from this baseline, especially significant increases in discovery data volumes, can be early warning signs of potential configuration churn issues. By proactively monitoring these reports and investigating any anomalies, you can identify and address potential churn causes before they escalate into major problems.

---

Managing configuration churn in Operations Manager is an ongoing process that requires vigilance, proactive monitoring, and a good understanding of discovery workflows and management pack behavior. By utilizing the techniques and tools described in this article, you can effectively detect, troubleshoot, and mitigate configuration churn, ensuring a stable and reliable monitoring environment.

If you have experienced configuration churn in your Operations Manager environment, or have further questions on this topic, please feel free to leave a comment below! Your experiences and insights can be valuable to the community.